Classification in hierarchical prediction domains

ABSTRACT

There is a need for solutions that classification solutions in hierarchical prediction domains. This need can be addressed by, for example, performing one or more online machine learning, co-occurrence analysis machine learning, structured fusion machine learning, and unstructured fusion machine learning. In one example, structured predictions inputs are processed in accordance with an online machine learning analysis to generate structurally hierarchical predictions and in accordance with a co-occurrence analysis machine learning analysis to generate structurally non-hierarchical predictions. Then, the structurally hierarchical predictions and the structurally non-hierarchical predictions in accordance with processed by a structured fusion model to generate structure-based predictions. Afterward, the structure-based predictions and non-structure-based predictions are processed in accordance with an unstructured fusion model to generate one or more unstructured-fused predictions.

BACKGROUND

Various embodiments of the present invention address technicalchallenges related to performing classification in hierarchicalprediction domains. Existing classification systems are ill-suited toefficiently and reliably perform classification in hierarchicalprediction domains. Various embodiments of the present invention addressthe shortcomings of the noted existing classification systems anddisclose various techniques for efficiently and reliably performingclassification in hierarchical prediction domains.

BRIEF SUMMARY

In general, embodiments of the present invention provide methods,apparatus, systems, computing devices, computing entities, and/or thelike for classification in hierarchical prediction domains. Certainembodiments utilize systems, methods, and computer program products thatenable entity sensitivity classification by using at least one of onlinemachine learning (ML) in hierarchical prediction domains, co-occurrenceanalysis in hierarchical prediction domains, fusion of structurallyhierarchical predictions and structurally non-hierarchical predictionsin hierarchical prediction domains, fusion of structure-basedpredictions and non-structure-based predictions in hierarchicalprediction domains, and Human Phenotype Ontology (HPO) predictions inthe hierarchical HPO label domain.

In accordance with one aspect, a method is provided. In one embodiment,the method comprises generating one or more structure-based predictionsbased at least in part on the one or more structured prediction inputsand using one or more structured machine learning models; generating oneor more non-structure-based predictions based at least in part on theone or more unstructured prediction inputs; obtaining access to anunstructured fusion machine learning model, wherein (i) the unstructuredfusion machine learning model is configured to perform an unstructuredfusion machine learning analysis based at least in part on the one ormore structure-based predictions and the one or more non-structure-basedpredictions to generate one or more unstructured-fused predictions; and(ii) the unstructured fusion machine learning analysis comprisesretraining the one or more structured machine learning models based atleast in part on the one or more non-structure-based predictions;performing the unstructured fusion machine learning analysis to generatethe one or more non-structure-based predictions; and generating, basedat least in part on the one or more unstructured-fused predictions, thepredictive output, wherein the predictive output indicates a selectednode of the plurality of prediction nodes for at least some of thestructurally hierarchical predictions.

In accordance with another aspect, a computer program product isprovided. The computer program product may comprise at least onecomputer-readable storage medium having computer-readable program codeportions stored therein, the computer-readable program code portionscomprising executable portions configured to generate one or morestructure-based predictions based at least in part on the one or morestructured prediction inputs and using one or more structured machinelearning models; generate one or more non-structure-based predictionsbased at least in part on the one or more unstructured predictioninputs; obtaining access to an unstructured fusion machine learningmodel, wherein (i) the unstructured fusion machine learning model isconfigured to perform an unstructured fusion machine learning analysisbased at least in part on the one or more structure-based predictionsand the one or more non-structure-based predictions to generate one ormore unstructured-fused predictions; and (ii) the unstructured fusionmachine learning analysis comprises retraining the one or morestructured machine learning models based at least in part on the one ormore non-structure-based predictions; perform the unstructured fusionmachine learning analysis to generate the one or morenon-structure-based predictions; and generate, based at least in part onthe one or more unstructured-fused predictions, the predictive output,wherein the predictive output indicates a selected node of the pluralityof prediction nodes for at least some of the structurally hierarchicalpredictions.

In accordance with yet another aspect, an apparatus comprising at leastone processor and at least one memory unit including computer programcode is provided. In one embodiment, the at least one memory unit andthe computer program code may be configured to, with the processor,cause the apparatus to generate one or more structure-based predictionsbased at least in part on the one or more structured prediction inputsand using one or more structured machine learning models; generate oneor more non-structure-based predictions based at least in part on theone or more unstructured prediction inputs; obtaining access to anunstructured fusion machine learning model, wherein (i) the unstructuredfusion machine learning model is configured to perform an unstructuredfusion machine learning analysis based at least in part on the one ormore structure-based predictions and the one or more non-structure-basedpredictions to generate one or more unstructured-fused predictions; and(ii) the unstructured fusion machine learning analysis comprisesretraining the one or more structured machine learning models based atleast in part on the one or more non-structure-based predictions;perform the unstructured fusion machine learning analysis to generatethe one or more non-structure-based predictions; and generate, based atleast in part on the one or more unstructured-fused predictions, thepredictive output, wherein the predictive output indicates a selectednode of the plurality of prediction nodes for at least some of thestructurally hierarchical predictions.

BRIEF DESCRIPTION OF THE DRAWINGS

Having thus described the invention in general terms, reference will nowbe made to the accompanying drawings, which are not necessarily drawn toscale, and wherein:

FIG. 1 provides an exemplary overview of an architecture that can beused to practice embodiments of the present invention.

FIG. 2 provides an example classification computing entity in accordancewith some embodiments discussed herein.

FIG. 3 provides an example external computing entity in accordance withsome embodiments discussed herein.

FIGS. 4A-4B provide data flow diagrams of example ensemble architecturesfor classification in a hierarchical prediction domain in accordancewith some embodiments discussed herein.

FIG. 5 is a flowchart diagram of an example process for training anonline ML model to perform predictive inferences related to ahierarchical prediction domain in accordance with some embodimentsdiscussed herein.

FIG. 6 is a flowchart diagram of an example process for generatinghierarchically-expanded training data in accordance with someembodiments discussed herein.

FIG. 7 provides an operational example of a raw training data object setin accordance with some embodiments discussed herein.

FIG. 8 provides an operational example of a hierarchical predictiondomain in accordance with some embodiments discussed herein.

FIG. 9 provides an operational example of a hierarchically-expandedtraining data object set in accordance with some embodiments discussedherein.

FIG. 10 is a flowchart diagram of an example process for generatinginitial weight values for a Follow-the-Regularized-Leader (FTRL) MLmodel in accordance with some embodiments discussed herein.

FIG. 11 is a flowchart diagram of an example process for storingappended training data objects in accordance with some embodimentsdiscussed herein.

FIG. 12 is a flowchart diagram of an example process for validating anFTRL ML model in accordance with some embodiments discussed herein.

FIG. 13 provides a flowchart diagram of an example process forgenerating prediction labels using a trained FTRL ML model and in ahierarchical prediction domain in accordance with some embodimentsdiscussed herein.

FIG. 14 provides an operational example of a predictive score dataobject in accordance with some embodiments discussed herein.

FIG. 15 is a flowchart diagram of an example process for training aco-occurrence analysis ML model in accordance with some embodimentsdiscussed herein.

FIG. 16 provides an operational example of a co-occurrence matrix inaccordance with some embodiments discussed herein.

FIG. 17 is a flowchart diagram of an example process for generating anormalized co-occurrence matrix in accordance with some embodimentsdiscussed herein.

FIG. 18 is a flowchart diagram of an example process for generatingpredictions using a trained co-occurrence analysis ML model inaccordance with some embodiments discussed herein.

FIG. 19 provides an operational example of a co-occurrence values setfor a training feature in accordance with some embodiments discussedherein.

FIG. 20 is a flowchart diagram of an example process for generatingpredictions based on structurally hierarchical predictions andstructurally non-hierarchically predictions in accordance with someembodiments discussed herein.

FIG. 21 provides an operational example of a structurally hierarchicalprediction set in accordance with some embodiments discussed herein.

FIG. 22 provides an operational example of a structurallynon-hierarchical prediction set in accordance with some embodimentsdiscussed herein.

FIG. 23 provides operational example of an up-weighting score generationdata structure in accordance with some embodiments discussed herein.

FIG. 24 provides an operational example of an up-weighting adjustmentdata structure in accordance with some embodiments discussed herein.

FIG. 25 provides a flowchart diagram of an example process forperforming an unstructured fusion of structure-based predictions andnon-structure-based predictions in accordance with some embodimentsdiscussed herein.

FIG. 26 is a flowchart diagram of an example process for generatingstructurally non-hierarchical predictions based on unstructuredprediction inputs in accordance with some embodiments discussed herein.

FIG. 27 provides an operational example of an unstructured input dataobject in accordance with some embodiments discussed herein.

FIG. 28 provides an operational example of a non-structure-basedprediction vector space in accordance with some embodiments discussedherein.

FIG. 29 is a flowchart diagram of an example process for generatingunstructured-fused predictions in accordance with some embodimentsdiscussed herein.

FIG. 30 is a flowchart diagram of an example process for performingHPO-based predictions in accordance with some embodiments discussedherein.

FIG. 31 provides an operational example of a patient-specific medicalcode record in accordance with some embodiments discussed herein.

FIG. 32 provides an operational example of a patient-specific phenotypicrecord in accordance with some embodiments discussed herein.

FIG. 33 provides an operational example of a cross patient holisticrecord in accordance with some embodiments discussed herein.

DETAILED DESCRIPTION

Various embodiments of the present invention now will be described morefully hereinafter with reference to the accompanying drawings, in whichsome, but not all embodiments of the inventions are shown. Indeed, theseinventions may be embodied in many different forms and should not beconstrued as limited to the embodiments set forth herein; rather, theseembodiments are provided so that this disclosure will satisfy applicablelegal requirements. The term “or” is used herein in both the alternativeand conjunctive sense, unless otherwise indicated. The terms“illustrative” and “exemplary” are used to be examples with noindication of quality level. Like numbers refer to like elementsthroughout. Moreover, while certain embodiments of the present inventionare described with reference to classification, one of ordinary skill inthe art will recognize that the disclosed concepts can be used toperform other types of data analysis.

I. OVERVIEW

Discussed herein methods, apparatus, systems, computing devices,computing entities, and/or the like for classification in hierarchicalprediction domains. As will be recognized, however, the disclosedconcepts can be used to perform any type of predictive data analysisand/or to perform classification in non-hierarchical prediction domains.

A. Some Technical Contributions of the Present Invention

Efficient and Reliable Classification in Hierarchical Prediction Domains

A hierarchical prediction domain is a prediction domain characterized byat least one hierarchical predictive relationship. In some embodiments,a hierarchical predictive relationship between two or more predictionnodes is a relationship between the prediction nodes based on which afirst prediction node has all of the attributes of each prediction nodefrom which it is deemed to be hierarchically dependent and one or moreadditional attributes. For example, given the hierarchical predictiverelationships C»A and C»B (where X»Y denotes that prediction node Y ishierarchically dependent on the prediction node X) in a particularprediction domain, a classification system may infer that predictionnodes A and B each have all of the attributes of prediction node C inaddition to one or more additional attributes.

Various embodiments of the present invention are directed toclassification in a hierarchical prediction domain by using at least oneof structured input data and unstructured data. Structured data mayrefer to data that can be divided into semantically-defined data objectsbased on a predefined format of the data. Examples of structured datainclude data defined using a Structured Query Language (SQL), datadefined using a file format language (such as the JavaScript ObjectNotation (JSON) language, a Comma-Separated Value (CSV) language, or anExtensible Markup Language (XML) language), and/or the like. In someembodiments, structured data used to perform classification may berepresented in a tabulated data format, e.g., a tabulated data formataccording to which rows represent entities and columns representattributes associated with various entities. In the healthcare context,structured data may include medical claims data, which may includeinformation associated with each medical claim (e.g., information abouttime of a medical operation associated with a medical claim, one or moreoperation codes associated with a medical claim, cost of a medicaloperation associated with a medical, and/or the like.) in a structuredformat. Unstructured data may refer to data that cannot be divided intosemantically-defined data objects based on a predefined format of thedata, e.g., free text data, hand-written note data, transcribed speechdata, etc. Examples of unstructured data include various types ofnatural language data, such as medical notes data which includes medicalnotes provided by a medical provider.

Thus, various embodiments of the present invention relate to performingpredictions related to prediction tasks characterized by ahierarchically complex prediction domain as well as a structurallycomplex input space. An example of a prediction task that present thecomplexities referred to herein is predicting HPO labels for a patientbased on medical data associated with the patient, such as medicalclaims data associated with the patient and medical notes dataassociated with the medical patient. The HPO label space, which providesa standardized vocabulary of phenotypic abnormalities associated withthousands of diseases, is an example of a hierarchical predictiondomain, as further described below. To perform HPO label predictionusing structured medical data and unstructured medical data, there is aneed for predictive analysis solutions that address the complexitiesassociated with the HPO label space as well as the complexitiesassociated with processing both structured medical data and unstructuredmedical data.

To perform predictions in a hierarchical prediction domains usingstructured input data and/or unstructured input data, variousembodiments of the present invention propose various arrangements of oneor more of the following ML models: an online ML model for processingstructured input data to generate structure-based predictions, aco-occurrence analysis ML model for processing structured input data togenerate structure-based predictions, a structured fusion ML model forcombining structure-based predictions, and an unstructured fusion MLmodel for combining structure-based predictions and non-structure-basedpredictions. In some embodiments, at least two of the mentioned MLmodels are organized in an ensemble architecture to generate a finalprediction based on predictions of the at least two ML models. In someembodiments, all of the mentioned ML models are organized in an ensemblearchitecture to generate a final prediction based on predictions of theat least two ML models.

Efficient and Reliable Online ML in Hierarchical Prediction Domains

Online learning is a method of ML in which a ML model is sequentiallyupdated over time based on incoming training data. During training, someonline learning algorithms aim to set parameters of a predictionfunction in a manner that minimizes measures of error betweenpredictions and existing training data labels, including new trainingdata labels and/or sequentially updated training data labels. Forexample, online learning algorithms are typically utilized to generaterecommendations for a user (e.g., promotional recommendations for auser), where the user reaction to the recommendation is in turn utilizedto update a ML model. In some online learning algorithms, a positiveuser reaction (e.g., a selection of a link corresponding to arecommendation) is used to change model parameters in a manner thatincreases a likelihood of future generation of the recommendation anddecreases a likelihood of future generation of other recommendations,while a negative user reaction (e.g., lack of selection of a linkcorresponding to a recommendation) is used to change model parameters ina manner that decreases a likelihood of future generation of therecommendation and increases a likelihood of future generation of otherrecommendations.

Hierarchical prediction domains present unique challenges for onlinelearning algorithms. When utilized to generate predictions related tohierarchical prediction domains, online learning algorithms shouldaccommodate hierarchical predictive relationships between variousprediction nodes in determining how to interpret incoming training data.Without applying appropriate operational adjustments that addresshierarchical nature of a relevant prediction domain, online learningalgorithms will require higher amounts of training data, will takelonger to train, and will once trained be less accurate and reliable.Because of those challenges, various existing online learning algorithmsare ill-suited for efficiently and reliably performing classification inrelation to hierarchical prediction domains.

Various embodiments of the present invention address efficiency andreliability challenges related to utilizing online learning algorithmsto generate predictions related to hierarchical prediction domains.According to one aspect that relates to improving efficiency andreliability of online learning in hierarchical prediction domains,various embodiments of the present invention eliminate a bias term usedto penalize lack of selection of a prediction node, as hierarchicalpredictive relationships complicate implications of such a lack ofselection for adjusting model parameters. For example, selection of aprediction node may have different implications for prediction nodesthat are dependent on the particular prediction node, prediction nodesfrom which the particular prediction node depends, and other predictionnodes without hierarchical relationships with the particular predictionnode. To address such complications, various embodiments of the presentinvention will not penalize lack of selection of a particular node whenadjusting parameters of a relevant ML model. In doing so, variousembodiments of the present invention address efficiency and reliabilitychallenges related to utilizing online learning algorithms to generatepredictions related to hierarchical prediction domains.

According to another aspect that relates to improving efficiency andreliability of online learning in hierarchical prediction domains,various embodiments of the present invention perform predictiveinferences by selecting prediction nodes having sufficiently highpredictive scores starting from dependent prediction nodes. In doing so,the mentioned embodiments of the present invention increase thelikelihood that prediction nodes having more detailed semanticimplications (e.g., more “meaningful” prediction nodes) will be selectedover prediction nodes having less detailed semantic associations, thusincreasing the reliability of the predictive analysis performed usingonline learning. For example, a prediction node associated with athoracolumbar scoliosis HPO label will have a higher chance of selectionthat a prediction node associated with a scoliosis HPO label, as theformer has a more meaningful semantic association than the former. Thiswill lead to generation of structurally hierarchical predictions whichhave greater predictive utility. Moreover, selection of predictionlabels in a hierarchical manner decreases the range of predictive scoresthat need to be analyzed during a predictive inference. This is because,according to various embodiments of the present invention, predictiveinference will halt if a requisite number of prediction nodes areselected inmost-dependent nodes. Such techniques have the addedadvantage of increasing efficiency of online learning in hierarchicalprediction domains by decreasing the range of prediction nodes whichneed to be traversed before a final prediction output is generated aspart of a particular predictive inference. Thus, by selecting predictionnodes having sufficiently high predictive scores starting from dependentprediction nodes, various embodiments of the present invention addressefficiency and reliability challenges related to utilizing onlinelearning algorithms to generate predictions related to hierarchicalprediction domains.

According to yet another aspect that relates to improving efficiency andreliability of online learning in hierarchical prediction domains,various embodiments of the present invention provide techniques forefficiently storing and retrieving training data. Online learningalgorithms face challenges related to efficiently storing and retrievingtraining data during training of relevant ML models. In the absence ofefficient solutions for storing and retrieving training data duringtraining of relevant ML models, many conventional online learningalgorithms are slow to train, which undermines the utility of suchalgorithms for predictive tasks that require real-time training and/orreal-time model updates. To address such challenges, various embodimentsof the present invention store training data entries in a highly sparsevector using a hashing mechanism. This aspect serves to increaseefficiency and reliability of online learning in all domains, includingin in hierarchical prediction domains. Thus, by storing training dataentries in a highly sparse vector using a hashing mechanism, variousembodiments of the present invention address efficiency and reliabilitychallenges related to utilizing online learning algorithms to generatepredictions related to hierarchical prediction domains.

Effective and Reliable Co-Occurrence Analysis in Hierarchical PredictionDomains

While online ML provides important insights about relationships amonghierarchical structures in a hierarchical prediction domains and has theflexibility of sequential updatability over time, other importantinsights can be inferred from analyzing statistical relations ofparticular features and particular prediction labels among trainingdata. However, given large amounts of training data, such statisticalanalyses may suffer from reliability drawbacks if they do not properlyaccommodate for factors that complicate conceptual predictive inferencesfrom numeric patterns. For example, trivially frequent correlations cancomplicate accurate and reliable conceptual inferences from statisticalcorrelations. As another example, markedly infrequent occurrences canalso complicate statistical analysis of predictive data in order toinfer conceptual notions that can facilitate effective classification.As a further example, conceptually obvious correlations may distortcross-data analyses of correlations between features and predictionlabels without contributing sufficient conceptual value to thepredictive inference process.

Because of such complexities associated with translation of numericpatterns to conceptual predictive frameworks, many existing statisticalML problems face substantial challenges when it comes to efficiently andreliably performing predictive inferences based on co-occurrence data.To address reliability concerns stemming from complexities associatedwith translation of numeric patterns to conceptual predictiveframeworks, many conventional statistical ML problems resort toexpensive training operations that undermine efficiency of ML solutionswithout sufficiently contributing to the reliability and accuracy of thepredictions performed by those ML solutions. Thus, there is a continuingtechnical need for efficient and reliable solutions for statistical MLin various classification domains, such as in hierarchical predictiondomains.

Various embodiments of the present invention address the efficiency andreliability challenges related to complexities associated withtranslation of numeric patterns to conceptual predictive frameworks. Forexample, various embodiments of the present invention provide innovativesolutions for both normalizing feature-label co-occurrence data andsignificance-based filtering of such co-occurrence data. Through thenoted techniques, various embodiments of the present invention providecomputationally efficient solutions that address complexities associatedwith translation of numeric patterns to conceptual predictiveframeworks, such as complexities associated with trivially frequentco-occurrences, complexities associated with mistakenly under-recordedco-occurrences, and complexities associated with conceptually obviousco-occurrences. Accordingly, by both normalizing feature-labelco-occurrence data and significance-based filtering of suchco-occurrence data, various embodiments of the present invention addresstechnical challenges related to efficiency and reliability ofstatistical ML solutions and improve efficiency and reliability ofvarious existing conventional statistical ML solutions. The resultingimprovements address efficiency and reliability of all statistical MLsolutions, including statistical ML solutions utilized in hierarchicalprediction domains. Thus, while aspects of the co-occurrence analysis MLmodels are described herein with respect to statistical ML solutionsutilized in hierarchical prediction domains, one of ordinary skill inthe art will recognize that the co-occurrence analysis ML models can beutilized to improve efficiency and reliability of all statistical MLsolutions, including statistical ML solutions utilized innon-hierarchical prediction domains.

In addition to improving efficiency and reliability of all statisticalML solutions, some aspects of the co-occurrence analysis ML modelsdescribed herein include important contributions to efficiency andreliability of ML in hierarchical prediction domains. In hierarchicalprediction domains, the presence of hierarchical relationships betweenprediction nodes in the output space complicates the task of inferring aprediction output based on prediction scores for various predictionnodes. On the one hand, the hierarchical relationships betweenprediction nodes in the output space provide important domaininformation that can facilitate efficient and reliable predictiveinferences. On the other hand, important predictive conclusions may beinferred from ignoring the hierarchical relationships, especially ininstances where the available hierarchical models do not capture all ofthe relevant information about conceptual relationships betweenprediction nodes and/or include potentially erroneous information aboutconceptual relationships between prediction nodes. Thus, there is acontinuing technical challenge associated with performing predictiveanalyses in a manner that takes into account both hierarchicalcomposition of the output space and cross-hierarchical composition ofthe output space.

Various embodiments of the present invention address the mentionedtechnical challenges associated with considering both hierarchicalcomposition of the output space and cross-hierarchical composition ofthe output space when performing classification in hierarchicalprediction domains. For example, various embodiments of the presentinvention relate to co-occurrence analysis ML models that generatestructurally non-hierarchical predictions. A structurallynon-hierarchical prediction may be a prediction determined withoutregard to a position of the corresponding prediction node in astructural hierarchy characterizing the hierarchical prediction domainthat includes the corresponding prediction node. For example, in someembodiments, a co-occurrence analysis ML model may be configured togenerate the one or more co-occurrence analysis predictions based onprediction values for prediction nodes regardless of whether theprediction nodes are deemed to be dependent prediction nodes in astructurally hierarchy characterizing the prediction domain associatedwith the co-occurrence analysis ML model. In at least some of thoseembodiments, the co-occurrence analysis ML model may generatepredictions that correspond to both most-dependent nodes andnon-most-dependent nodes in the structural hierarchically characterizingthe prediction domain associated with the online ML model.

By generating structurally non-hierarchical predictions, variousembodiments of the present invention provide predictions that areagnostic to the hierarchical composition of the prediction output space.When used in combination and/or in fusion with structurally hierarchicalpredictions (e.g., online learning predictions generated by an online MLmodel), such predictions can provide important cross-hierarchicalconceptual inferences that can in turn facilitate efficient andeffective classification in conceptually hierarchical domains. Thus, bygenerating structurally non-hierarchical predictions that can in turn beused in combination and/or in fusion with structurally hierarchicalpredictions, various embodiments of the present invention addresstechnical challenges related to accounting for both hierarchicalcomposition of the output space and cross-hierarchical composition ofthe output space when performing classification in hierarchicalprediction domains. In doing so, various embodiments of the presentinvention make important technical contributions to efficiency andreliability of classification in hierarchical prediction domains, suchas in classification in an HPO prediction domain.

Efficient and Reliable Fusion of Structurally Hierarchical Predictionsand Structurally Non-Hierarchical Predictions

As discussed above with reference to co-occurrence analysis ML models,in hierarchical prediction domains, the presence of hierarchicalrelationships between prediction nodes in the output space complicatesthe task of inferring a prediction output based on prediction scores forvarious prediction nodes. On the one hand, the hierarchicalrelationships between prediction nodes in the output space provideimportant domain information that can facilitate efficient and reliablepredictive inferences. On the other hand, important predictiveconclusions may be inferred from ignoring the hierarchicalrelationships, especially in instances where the available hierarchicalmodels do not capture all of the relevant information about conceptualrelationships between prediction nodes and/or include potentiallyerroneous information about conceptual relationships between predictionnodes. Thus, there is a continuing technical challenge associated withperforming predictive analyses in a manner that considers bothhierarchical composition of the output space and cross-hierarchicalcomposition of the output space.

Various embodiments of the present invention address the mentionedtechnical challenges associated with considering both hierarchicalcomposition of the output space and cross-hierarchical composition ofthe output space when performing classification in hierarchicalprediction domains. For example, various embodiments of the presentinvention relate to co-occurrence analysis ML models that generatestructurally non-hierarchical predictions. A structurallynon-hierarchical prediction may be a prediction determined withoutregard to a position of the corresponding prediction node in astructural hierarchy characterizing the hierarchical prediction domainthat includes the corresponding prediction node. For example, in someembodiments, a co-occurrence analysis ML model may be configured togenerate the one or more co-occurrence analysis predictions based onprediction values for prediction nodes regardless of whether theprediction nodes are deemed to be dependent prediction nodes in astructurally hierarchy characterizing the prediction domain associatedwith the co-occurrence analysis ML model. In at least some of thoseembodiments, the co-occurrence analysis ML model may generatepredictions that correspond to both most-dependent nodes andnon-most-dependent nodes in the structural hierarchically characterizingthe prediction domain associated with the online ML model.

By generating structurally non-hierarchical predictions, variousembodiments of the present invention provide predictions that areagnostic to the hierarchical composition of the prediction output space.Such structurally non-hierarchical predictions can in turn be used incombination and/or in fusion with structurally hierarchical predictions,such as structurally hierarchical predictions generated by an onlinelearning unit 111. When structurally non-hierarchical predictions areused in combination and/or in fusion with structurally hierarchicalpredictions to generate structure-based predictions, suchstructured-fused predictions can provide important cross-hierarchicalconceptual inferences that can in turn facilitate efficient andeffective classification in conceptually hierarchical domains. Variousembodiments of the present invention provide efficient and reliabletechniques for fusing structurally hierarchical predictions andstructurally non-hierarchical predictions. Such solutions make importanttechnical contributions to classification models in hierarchicalprediction domains, as they enable such models to utilize bothpredictive insights provided by hierarchical relationships of the outputspace and predictive insights provided without taking hierarchicalrelationships among training data into account. In doing so, variousembodiments of the present invention address key challenges related toefficiency and reliability of classification in hierarchical predictiondomains, such as the efficiency and reliability of HPO label prediction.

Efficient and Reliable Fusion of Structure-Based Predictions andNon-Structure-Based Predictions

Various embodiments of the present invention are directed toclassification in a hierarchical prediction domain by using at least oneof structured input data and unstructured data. Structured data mayrefer to data that can be divided into semantically-defined data objectsbased on a predefined format of the data. Examples of structured datainclude data defined using a Structured Query Language (SQL), datadefined using a file format language (such as the JavaScript ObjectNotation (JSON) language, a Comma-Separated Value (CSV) language, or anExtensible Markup Language (XML) language), and/or the like. In someembodiments, structured data used to perform classification may berepresented in a tabulated data format, e.g., a tabulated data formataccording to which rows represent entities and columns representattributes associated with various entities. In the healthcare context,structured data may include medical claims data, which may includeinformation associated with each medical claim (e.g., information abouttime of a medical operation associated with a medical claim, one or moreoperation codes associated with a medical claim, cost of a medicaloperation associated with a medical, and/or the like.) in a structuredformat. Unstructured data may refer to data that cannot be divided intosemantically-defined data objects based on a predefined format of thedata, e.g., handwritten note data, transcribed speech data, image data,etc. Examples of unstructured data include various types of naturallanguage data, such as medical notes data which includes medical notesprovided by a medical provider.

Both structured data and unstructured data provide valuable predictiveinsights for predictive analysis tasks, e.g., for predictive analysistasks related to hierarchical prediction domains. For example,structured data can provide important insights about statisticaldistribution of features and prediction labels as well as sequentialchange of correlations between features and prediction labels over time.In some cases, structured data can provide insights that are out of thereach of semantically-unsophisticated and/or primarily-lexical naturallanguage processing algorithms for processing unstructured data. On theother hand, when properly analyzed (e.g., when analyzed usingsemantically-sophisticated synonym-based natural language processingalgorithms), unstructured data can provide a strong source of predictiveinsights about a predictive task associated with a hierarchicalprediction domain.

Despite the complimentary utility of structured data and unstructureddata in providing predictive insights relevant to classification inhierarchical prediction domains, the problem of efficiently andeffectively integrating predictions derived from structured data (e.g.,structure-based predictions) and predictions derived from unstructureddata (e.g., non-structure-based predictions) is a non-trivial problemfrom a technical standpoint. Indeed, many conventional classificationsolutions fail to efficiently and reliably integrate structure-basedpredictions and non-structure-based predictions to generate predictiveoutputs. For example, a naive combination of particular structure-basedpredictions and non-structure-based predictions fails to properlyappreciate the reciprocal implications of structure-based predictionsand non-structure-based predictions for improving models utilized togenerate each other. Indeed, one innovative aspect of the presentinvention relates to techniques for efficiently and reliably integratingstructure-based predictions and non-structure-based predictions in amanner that causes at least one of the noted sets of predictions toprovide feedback to a model utilized to generate the other.

Accordingly, various embodiments of the present invention addresstechnical challenges related to efficient and reliable fusion ofstructure-based predictions and non-structure-based predictions byutilizing at least one of the noted sets of predictions to providefeedback to a model utilized to generate the other. For example, in someembodiments, non-structure-based predictions are used as ground-truthdata to retrain one or more ML models utilized to generatestructure-based predictions, e.g., one or more of an online ML model anda co-occurrence analysis ML model,. Through this and similar techniques,various embodiments of the present invention enable feedback-loopmechanism relationships between structure-based predictions andnon-structure-based predictions which serve to render the modelsutilized to generate at least one of the structure-based predictions andthe non-structure-based predictions more efficient (both in terms oftraining efficiency and in terms of inference efficiency) as well asmore reliable. Thus, by utilizing at least one of the noted sets ofpredictions to provide feedback to a model utilized to generate theother, various embodiments of the present invention address technicalchallenges related to efficient and reliable fusion of structure-basedpredictions and non-structure-based predictions and make substantialtechnical improvements to conventional solutions for classification,such as conventional solutions for classification in hierarchicalprediction domains.

Efficient and Reliable HPO Label Prediction

HPO label prediction is an example of a prediction task related to ahierarchical prediction domain. As discussed above and further describedbelow, the hierarchical prediction domains such as the HPO label domainpresent significant problems for various classification approaches.Examples of these challenges include challenges associated withstructural complexity of the output space of such hierarchicalprediction domains as well as challenges associated with complexity ofinput space of hierarchical prediction domains. Accordingly, to performHPO label prediction using structured medical data and unstructuredmedical data, there is a need for predictive analysis solutions thataddress the complexities associated with the HPO label space as well asthe complexities associated with processing both structured medical dataand unstructured medical data.

To perform predictions in a hierarchical prediction domains usingstructured input data and/or unstructured input data, variousembodiments of the present invention propose various arrangements of oneor more of the following ML models: an online ML model for processingstructured input data to generate structure-based predictions, aco-occurrence analysis ML model for processing structured input data togenerate structure-based predictions, a structured fusion ML model forcombining structure-based predictions, and an unstructured fusion MLmodel for combining structure-based predictions and non-structure-basedpredictions. In some embodiments, at least two of the mentioned MLmodels are organized in an ensemble architecture to generate a finalprediction based on predictions of the at least two ML models. In someembodiments, all of the mentioned ML models are organized in an ensemblearchitecture to generate a final prediction based on predictions of theat least two ML models. Such ensemble architectures provide efficientand reliable solutions for classification in hierarchical predictiondomain, such as for HPO label prediction in relation to the HPO labeldomain.

In addition, hierarchical prediction domains like the HPO domain presentunique challenges for online learning algorithms. When utilized togenerate predictions related to hierarchical prediction domains, onlinelearning algorithms should accommodate hierarchical predictiverelationships between various prediction nodes in determining how tointerpret incoming training data. Without applying appropriateoperational adjustments that address hierarchical nature of a relevantprediction domain, online learning algorithms will require higheramounts of training data, will take longer to train, and will oncetrained be less accurate and reliable. Because of those challenges,various existing online learning algorithms are ill-suited forefficiently and reliably performing classification in relation tohierarchical prediction domains.

Various embodiments of the present invention address efficiency andreliability challenges related to utilizing online learning algorithmsto generate predictions related to hierarchical prediction domains.According to one aspect that relates to improving efficiency andreliability of online learning in hierarchical prediction domains,various embodiments of the present invention eliminate a bias term usedto penalize lack of selection of a prediction node, as hierarchicalpredictive relationships complicate implications of such a lack ofselection for adjusting model parameters. For example, selection of aprediction node may have different implications for prediction nodesthat are dependent on the particular prediction node, prediction nodesfrom which the particular prediction node depends, and other predictionnodes without hierarchical relationships with the particular predictionnode. To address such complications, various embodiments of the presentinvention will not penalize lack of selection of a particular node whenadjusting parameters of a relevant ML model. In doing so, variousembodiments of the present invention address efficiency and reliabilitychallenges related to utilizing online learning algorithms to generatepredictions related to hierarchical prediction domains, such generatingHPO label predictions related to the HPO label domain.

Next, some aspects of the co-occurrence analysis ML models describedherein include important contributions to efficiency and reliability ofML in hierarchical prediction domains, such as the HPO predictiondomain. In hierarchical prediction domains, the presence of hierarchicalrelationships between prediction nodes in the output space complicatesthe task of inferring a prediction output based on prediction scores forvarious prediction nodes. On the one hand, the hierarchicalrelationships between prediction nodes in the output space provideimportant domain information that can facilitate efficient and reliablepredictive inferences. On the other hand, important predictiveconclusions may be inferred from ignoring the hierarchicalrelationships, especially in instances where the available hierarchicalmodels do not capture all of the relevant information about conceptualrelationships between prediction nodes and/or include potentiallyerroneous information about conceptual relationships between predictionnodes. Thus, there is a continuing technical challenge associated withperforming predictive analyses in a manner that takes into account bothhierarchical composition of the output space and cross-hierarchicalcomposition of the output space.

Various embodiments of the present invention address the mentionedtechnical challenges associated with considering both hierarchicalcomposition of the output space and cross-hierarchical composition ofthe output space when performing classification in hierarchicalprediction domains. For example, various embodiments of the presentinvention relate to co-occurrence analysis ML models that generatestructurally non-hierarchical predictions. A structurallynon-hierarchical prediction may be a prediction determined withoutregard to a position of the corresponding prediction node in astructural hierarchy characterizing the hierarchical prediction domainthat includes the corresponding prediction node. By generatingstructurally non-hierarchical predictions, various embodiments of thepresent invention provide predictions that are agnostic to thehierarchical composition of the prediction output space. When used incombination and/or in fusion with structurally hierarchical predictions(e.g., online learning predictions generated by an online ML model),such predictions can provide important cross-hierarchical conceptualinferences that can in turn facilitate efficient and effectiveclassification in conceptually hierarchical domains. Thus, by generatingstructurally non-hierarchical predictions that can in turn be used incombination and/or in fusion with structurally hierarchical predictions,various embodiments of the present invention address technicalchallenges related to accounting for both hierarchical composition ofthe output space and cross-hierarchical composition of the output spacewhen performing classification in hierarchical prediction domains. Indoing so, various embodiments of the present invention make importanttechnical contributions to efficiency and reliability of classificationin hierarchical prediction domains, such as in classification in an HPOprediction domain and with respect to the HPO label predictionpredictive task.

Furthermore, hierarchical prediction domains like the HPO predictiondomain present challenges related to fusion of structurally hierarchicalpredictions and non-structurally hierarchical predictions. By generatingstructurally non-hierarchical predictions, various embodiments of thepresent invention provide predictions that are agnostic to thehierarchical composition of the prediction output space. Suchstructurally non-hierarchical predictions can in turn be used incombination and/or in fusion with structurally hierarchical predictions,such as structurally hierarchical predictions generated by an onlinelearning unit 111. When structurally non-hierarchical predictions areused in combination and/or in fusion with structurally hierarchicalpredictions are used to generate structure-based predictions, suchstructured-fused predictions can provide important cross-hierarchicalconceptual inferences that can in turn facilitate efficient andeffective classification in conceptually hierarchical domains.

Various embodiments of the present invention provide efficient andreliable techniques for fusing structurally hierarchical predictions andstructurally non-hierarchical predictions. Such solutions make importanttechnical contributions to classification models in hierarchicalprediction domains, as they enable such models to utilize bothpredictive insights provided by hierarchical relationships of the outputspace and predictive insights provided without taking hierarchicalrelationships among training data into account. In doing so, variousembodiments of the present invention address key challenges related toefficiency and reliability of classification in hierarchical predictiondomains, such as the efficiency and reliability of HPO label prediction.

Moreover, hierarchical prediction domains like the HPO domain presentchallenges related to fusion of structure-based predictions andnon-structure-based predictions. Both structured data and unstructureddata provide valuable predictive insights for predictive analysis tasks,e.g., for predictive analysis tasks related to hierarchical predictiondomains. However, despite the complimentary utility of structured dataand unstructured data in providing predictive insights relevant toclassification in hierarchical prediction domains, the problem ofefficiently and effectively integrating predictions derived fromstructured data (i.e., structure-based predictions) and predictionsderived from unstructured data (i.e., non-structure-based predictions)is a non-trivial problem from a technical standpoint. Indeed, manyconventional classification solutions fail to efficiently and reliablyintegrate structure-based predictions and non-structure-basedpredictions to generate predictive outputs. For example, a naivecombination of particular structure-based predictions andnon-structure-based predictions fails to properly appreciate thereciprocal implications of structure-based predictions andnon-structure-based predictions for improving models utilized togenerate each other. Indeed, one innovative aspect of the presentinvention relates to techniques for efficiently and reliably integratingstructure-based predictions and non-structure-based predictions in amanner that causes at least one of the noted sets of predictions toprovide feedback to a model utilized to generate the other.

Accordingly, various embodiments of the present invention addresstechnical challenges related to efficient and reliable fusion ofstructure-based predictions and non-structure-based predictions byutilizing at least one of the noted sets of predictions to providefeedback to a model utilized to generate the other. For example, in someembodiments, non-structure-based predictions are used as ground-truthdata to retrain one or more ML models utilized to generatestructure-based predictions, e.g., one or more of an online ML model anda co-occurrence analysis ML model. Through this and similar techniques,various embodiments of the present invention enable feedback-loopmechanism relationships between structure-based predictions andnon-structure-based predictions which serve to render the modelsutilized to generate at least one of the structure-based predictions andthe non-structure-based predictions more efficient (both in terms oftraining efficiency and in terms of inference efficiency) as well asmore reliable. Thus, by utilizing at least one of the noted sets ofpredictions to provide feedback to a model utilized to generate theother, various embodiments of the present invention address technicalchallenges related to efficient and reliable fusion of structure-basedpredictions and non-structure-based predictions and make substantialtechnical improvements to conventional solutions for classification,such as conventional solutions for classification in hierarchicalprediction domains.

II. COMPUTER PROGRAM PRODUCTS, METHODS, AND COMPUTING ENTITIES

Embodiments of the present invention may be implemented in various ways,including as computer program products that comprise articles ofmanufacture. Such computer program products may include one or moresoftware components including, for example, software objects, methods,data structures, or the like. A software component may be coded in anyof a variety of programming languages. An illustrative programminglanguage may be a lower-level programming language such as an assemblylanguage associated with a particular hardware architecture and/oroperating system platform. A software component comprising assemblylanguage instructions may require conversion into executable machinecode by an assembler prior to execution by the hardware architectureand/or platform. Another example programming language may be ahigher-level programming language that may be portable across multiplearchitectures. A software component comprising higher-level programminglanguage instructions may require conversion to an intermediaterepresentation by an interpreter or a compiler prior to execution.

Other examples of programming languages include, but are not limited to,a macro language, a shell or command language, a job control language, ascript language, a database query or search language, and/or a reportwriting language. In one or more example embodiments, a softwarecomponent comprising instructions in one of the foregoing examples ofprogramming languages may be executed directly by an operating system orother software component without having to be first transformed intoanother form. A software component may be stored as a file or other datastorage construct. Software components of a similar type or functionallyrelated may be stored together such as, for example, in a particulardirectory, folder, or library. Software components may be static (e.g.,pre-established or fixed) or dynamic (e.g., created or modified at thetime of execution).

A computer program product may include a non-transitorycomputer-readable storage medium storing applications, programs, programmodules, scripts, source code, program code, object code, byte code,compiled code, interpreted code, machine code, executable instructions,and/or the like (also referred to herein as executable instructions,instructions for execution, computer program products, program code,and/or similar terms used herein interchangeably). Such non-transitorycomputer-readable storage media include all computer-readable media(including volatile and non-volatile media).

In one embodiment, a non-volatile computer-readable storage medium mayinclude a floppy disk, flexible disk, hard disk, solid-state storage(SSS) (e.g., a solid state drive (SSD), solid state card (SSC), solidstate module (SSM), enterprise flash drive, magnetic tape, or any othernon-transitory magnetic medium, and/or the like. A non-volatilecomputer-readable storage medium may also include a punch card, papertape, optical mark sheet (or any other physical medium with patterns ofholes or other optically recognizable indicia), compact disc read onlymemory (CD-ROM), compact disc-rewritable (CD-RW), digital versatile disc(DVD), Blu-ray disc (BD), any other non-transitory optical medium,and/or the like. Such a non-volatile computer-readable storage mediummay also include read-only memory (ROM), programmable read-only memory(PROM), erasable programmable read-only memory (EPROM), electricallyerasable programmable read-only memory (EEPROM), flash memory (e.g.,Serial, NAND, NOR, and/or the like), multimedia memory cards (MMC),secure digital (SD) memory cards, SmartMedia cards, CompactFlash (CF)cards, Memory Sticks, and/or the like. Further, a non-volatilecomputer-readable storage medium may also include conductive-bridgingrandom access memory (CBRAIVI), phase-change random access memory(PRAM), ferroelectric random-access memory (FeRAM), non-volatilerandom-access memory (NVRAIVI), magnetoresistive random-access memory(MRAM), resistive random-access memory (RRAIVI),Silicon-Oxide-Nitride-Oxide-Silicon memory (SONOS), floating junctiongate random access memory (FJG RAM), Millipede memory, racetrack memory,and/or the like.

In one embodiment, a volatile computer-readable storage medium mayinclude random access memory (RAM), dynamic random access memory (DRAM),static random access memory (SRAM), fast page mode dynamic random accessmemory (FPM DRAM), extended data-out dynamic random access memory (EDODRAM), synchronous dynamic random access memory (SDRAM), double datarate synchronous dynamic random access memory (DDR SDRAM), double datarate type two synchronous dynamic random access memory (DDR2 SDRAM),double data rate type three synchronous dynamic random access memory(DDR3 SDRAM), Rambus dynamic random access memory (RDRAM), TwinTransistor RAM (TTRAM), Thyristor RAM (T-RAM), Zero-capacitor (Z-RAM),Rambus in-line memory module (RIMM), dual in-line memory module (DIMM),single in-line memory module (SIMM), video random access memory (VRAM),cache memory (including various levels), flash memory, register memory,and/or the like. It will be appreciated that where embodiments aredescribed to use a computer-readable storage medium, other types ofcomputer-readable storage media may be substituted for or used inaddition to the computer-readable storage media described above.

As should be appreciated, various embodiments of the present inventionmay also be implemented as methods, apparatus, systems, computingdevices, computing entities, and/or the like. As such, embodiments ofthe present invention may take the form of an apparatus, system,computing device, computing entity, and/or the like executinginstructions stored on a computer-readable storage medium to performcertain steps or operations. Thus, embodiments of the present inventionmay also take the form of an entirely hardware embodiment, an entirelycomputer program product embodiment, and/or an embodiment that comprisescombination of computer program products and hardware performing certainsteps or operations. Embodiments of the present invention are describedbelow with reference to block diagrams and flowchart illustrations.Thus, it should be understood that each block of the block diagrams andflowchart illustrations may be implemented in the form of a computerprogram product, an entirely hardware embodiment, a combination ofhardware and computer program products, and/or apparatus, systems,computing devices, computing entities, and/or the like carrying outinstructions, operations, steps, and similar words used interchangeably(e.g., the executable instructions, instructions for execution, programcode, and/or the like) on a computer-readable storage medium forexecution. For example, retrieval, loading, and execution of code may beperformed sequentially such that one instruction is retrieved, loaded,and executed at a time. In some exemplary embodiments, retrieval,loading, and/or execution may be performed in parallel such thatmultiple instructions are retrieved, loaded, and/or executed together.Thus, such embodiments can produce specifically-configured machinesperforming the steps or operations specified in the block diagrams andflowchart illustrations. Accordingly, the block diagrams and flowchartillustrations support various combinations of embodiments for performingthe specified instructions, operations, or steps.

III. EXEMPLARY SYSTEM ARCHITECTURE

The architecture 100 includes one or more external computing entities102 that interact with a classification system 101 via a communicationnetwork (not shown). The classification system 101 includes a storagesubsystem 108 and a classification computing entity 106. Each computingentity, computing subsystem, and/or computing system in the architecture100 may include any suitable network server and/or other type ofprocessing device. The communication network may include any wired orwireless communication network including, for example, a wired orwireless local area network (LAN), personal area network (PAN),metropolitan area network (MAN), wide area network (WAN), or the like,as well as any hardware, software and/or firmware required to implementit (such as, e.g., network routers, and/or the like).

In some embodiments, the architecture 100 is configured to enable theexternal computing entities 102 to provide prediction inputs to theclassification system 101 and, in response, receive predictionsgenerated based on the prediction inputs. For example, a particularexternal computing entity 102 may provide a request for an HPO labelprediction for a patient, where the request may include data associatedwith the patient (e.g., personal attribute data associated with thepatient, medical codes associated with the medical history of thepatient, and/or the like.). The classification system 101 is configuredto generate the requested HPO label predictions and provide thegenerated HPO label predictions to the particular external computingentity 102.

The classification computing entity 106 includes a model generation unit115 configured to train at least one ML model utilized by theclassification computing entity 106 to perform predictions. Examples ofmodels trained by the model generation unit 115 may include an online MLmodel and a co-occurrence analysis ML model. The classificationcomputing entity 106 further includes an online learning unit 111configured to apply the online ML model to a particular prediction inputto generate a corresponding co-occurrence analysis prediction.

In some embodiments, at least some of the ML models utilized by theclassification computing entity 106 to perform predictions may producestructurally hierarchical predictions while at least some other MLmodels utilized by the classification computing entity 106 to performpredictions may produce non-structurally hierarchical predictions. Insome of those embodiments, a structured fusion unit 113 of theclassification computing entity 106 is configured to combine at leastone structurally hierarchical prediction and at least one structurallynon-hierarchical prediction to generate a structure-based prediction.

In some embodiments, at least some of the ML models utilized by theclassification computing entity 106 to perform predictions may utilizestructured data to produce structure-based predictions, while at leastsome other ML models utilized by the classification computing entity 106to perform predictions may utilize unstructured data to producenon-structure-based predictions. In some of those embodiments, anunstructured fusion unit 114 of the classification computing entity 106is configured to combine at least one structure-based prediction and atleast one non-structure-based prediction to generate anunstructured-fused prediction.

The classification computing entity 106 may further include a systeminteraction unit 116 configured to generate predictions based on atleast one of the ML models utilized by the classification computingentity 106, for example based on at least one of the online ML model,the co-occurrence analysis ML model, the structured fusion ML model, andthe unstructured fusion ML model. In some embodiments, the systeminteraction unit 116 is configured to utilize one or more ML models togenerate HPO labels. In some of those embodiments, the systeminteraction unit 116 is further configured to generate HPO-basedpredictions and/or HPO-based data reports, such as standardized genomictesting frameworks, integrated genomic record repositories, andprecision medicine analytics. In some embodiments, the systeminteraction unit 116 is configured to perform one or more actions (e.g.,transmission of communications, activation of alerts, automaticscheduling of appointments, and/or the like.) based on the predicted HPOlabels and/or based on the generated HPO-based predictions.

The storage subsystem 108 may include one or more storage units, such asmultiple distributed storage units that are connected through a computernetwork. Each storage unit in the storage subsystem 108 may store atleast one of one or more data assets and/or one or more data about thecomputed properties of one or more data assets. Moreover, each storageunit in the storage subsystem 108 may include one or more non-volatilestorage or memory media including but not limited to hard disks, ROM,PROM, EPROM, EEPROM, flash memory, MMCs, SD memory cards, Memory Sticks,CBRAM, PRAM, FeRAM, NVRAM, MRAM, RRAM, SONOS, FJG RAM, Millipede memory,racetrack memory, and/or the like. The storage subsystem 108 may includestructured input data 121, unstructured input data 122, model definitiondata 123 (e.g., data defining at least one parameter and/or at least onehyper-parameter of at least one ML model utilized by the classificationcomputing entity 106), and raw training data 124 (e.g., data utilized bythe model generation unit 115 to train at least one ML model utilized bythe classification computing entity 106, such as hierarchicallynon-expanded data utilized by the model generation unit 115 to train atleast one ML model utilized by the classification computing entity 106).

B. Exemplary Classification Computing Entity

FIG. 2 provides a schematic of a classification computing entity 106according to one embodiment of the present invention. In general, theterms computing entity, computer, entity, device, system, and/or similarwords used herein interchangeably may refer to, for example, one or morecomputers, computing entities, desktops, mobile phones, tablets,phablets, notebooks, laptops, distributed systems, kiosks, inputterminals, servers or server networks, blades, gateways, switches,processing devices, processing entities, set-top boxes, relays, routers,network access points, base stations, the like, and/or any combinationof devices or entities adapted to perform the functions, operations,and/or processes described herein. Such functions, operations, and/orprocesses may include, for example, transmitting, receiving, operatingon, processing, displaying, storing, determining, creating/generating,monitoring, evaluating, comparing, and/or similar terms used hereininterchangeably. In one embodiment, these functions, operations, and/orprocesses can be performed on data, content, information, and/or similarterms used herein interchangeably.

As indicated, in one embodiment, the classification computing entity 106may also include one or more communications interfaces 220 forcommunicating with various computing entities, such as by communicatingdata, content, information, and/or similar terms used hereininterchangeably that can be transmitted, received, operated on,processed, displayed, stored, and/or the like.

As shown in FIG. 2, in one embodiment, the classification computingentity 106 may include or be in communication with one or moreprocessing elements 205 (also referred to as processors, processingcircuitry, and/or similar terms used herein interchangeably) thatcommunicate with other elements within the classification computingentity 106 via a bus, for example. As will be understood, the processingelement 205 may be embodied in a number of different ways. For example,the processing element 205 may be embodied as one or more ComplexPogrammable Logic Devices (CPLDs), microprocessors, multi-coreprocessors, coprocessing entities, Application-Specific Instruction-SetProcessors (ASIPs), microcontrollers, and/or controllers. Further, theprocessing element 205 may be embodied as one or more other processingdevices or circuitry. The term circuitry may refer to an entirelyhardware embodiment or a combination of hardware and computer programproducts. Thus, the processing element 205 may be embodied as integratedcircuits, Application Specific Integrated Circuits (ASICs), FieldProgrammable Gate Arrays (FPGAs), Programmable Logic Arrays (PLAs),hardware accelerators, other circuitry, and/or the like. As willtherefore be understood, the processing element 205 may be configuredfor a particular use or configured to execute instructions stored involatile or non-volatile media or otherwise accessible to the processingelement 205. As such, whether configured by hardware or computer programproducts, or by a combination thereof, the processing element 205 may becapable of performing steps or operations according to embodiments ofthe present invention when configured accordingly.

In one embodiment, the classification computing entity 106 may furtherinclude or be in communication with non-volatile media (also referred toas non-volatile storage, memory, memory storage, memory circuitry and/orsimilar terms used herein interchangeably). In one embodiment, thenon-volatile storage or memory may include one or more non-volatilestorage or memory media 210, including but not limited to hard disks,ROM, PROM, EPROM, EEPROM, flash memory, MMCs, SD memory cards, MemorySticks, CBRAM, PRAM, FeRAM, NVRAM, MRAM, RRAM, SONOS, FJG RAM, Millipedememory, racetrack memory, and/or the like. As will be recognized, thenon-volatile storage or memory media may store databases, databaseinstances, database management systems, data, applications, programs,program modules, scripts, source code, object code, byte code, compiledcode, interpreted code, machine code, executable instructions, and/orthe like. The term database, database instance, database managementsystem, and/or similar terms used herein interchangeably may refer to acollection of records or data that is stored in a computer-readablestorage medium using one or more database models, such as a hierarchicaldatabase model, network model, relational model, entity-relationshipmodel, object model, document model, semantic model, graph model, and/orthe like.

In one embodiment, the classification computing entity 106 may furtherinclude or be in communication with volatile media (also referred to asvolatile storage, memory, memory storage, memory circuitry and/orsimilar terms used herein interchangeably). In one embodiment, thevolatile storage or memory may also include one or more volatile storageor memory media 215, including but not limited to RAM, DRAM, SRAM, FPMDRAM, EDO DRAM, SDRAM, DDR SDRAM, DDR2 SDRAM, DDR3 SDRAM, RDRAM, TTRAM,T-RAM, Z-RAM, RIMM, DIMM, SIMM, VRAM, cache memory, register memory,and/or the like. As will be recognized, the volatile storage or memorymedia may be used to store at least portions of the databases, databaseinstances, database management systems, data, applications, programs,program modules, scripts, source code, object code, byte code, compiledcode, interpreted code, machine code, executable instructions, and/orthe like being executed by, for example, the processing element 205.Thus, the databases, database instances, database management systems,data, applications, programs, program modules, scripts, source code,object code, byte code, compiled code, interpreted code, machine code,executable instructions, and/or the like may be used to control certainaspects of the operation of the classification computing entity 106 withthe assistance of the processing element 205 and operating system.

As indicated, in one embodiment, the classification computing entity 106may also include one or more communications interfaces 220 forcommunicating with various computing entities, such as by communicatingdata, content, information, and/or similar terms used hereininterchangeably that can be transmitted, received, operated on,processed, displayed, stored, and/or the like. Such communication may beexecuted using a wired data transmission protocol, such as fiberdistributed data interface (FDDI), digital subscriber line (DSL),Ethernet, asynchronous transfer mode (ATM), frame relay, data over cableservice interface specification (DOCSIS), or any other wiredtransmission protocol. Similarly, the classification computing entity106 may be configured to communicate via wireless external communicationnetworks using any of a variety of protocols, such as general packetradio service (GPRS), Universal Mobile Telecommunications System (UMTS),Code Division Multiple Access 2000 (CDMA2000), CDMA2000 1× (1×RTT),Wideband Code Division Multiple Access (WCDMA), Global System for MobileCommunications (GSM), Enhanced Data rates for GSM Evolution (EDGE), TimeDivision-Synchronous Code Division Multiple Access (TD-SCDMA), Long TermEvolution (LTE), Evolved Universal Terrestrial Radio Access Network(E-UTRAN), Evolution-Data Optimized (EVDO), High Speed Packet Access(HSPA), High-Speed Downlink Packet Access (HSDPA), IEEE 802.11 (Wi-Fi),Wi-Fi Direct, 802.16 (WiMAX), ultra-wideband (UWB), infrared (IR)protocols, near field communication (NFC) protocols, Wibree, Bluetoothprotocols, wireless universal serial bus (USB) protocols, and/or anyother wireless protocol.

Although not shown, the classification computing entity 106 may includeor be in communication with one or more input elements, such as akeyboard input, a mouse input, a touch screen/display input, motioninput, movement input, audio input, pointing device input, joystickinput, keypad input, and/or the like. The classification computingentity 106 may also include or be in communication with one or moreoutput elements (not shown), such as audio output, video output,screen/display output, motion output, movement output, and/or the like.

C. Exemplary External Computing Entity

FIG. 3 provides an illustrative schematic representative of an externalcomputing entity 102 that can be used in conjunction with embodiments ofthe present invention. In general, the terms device, system, computingentity, entity, and/or similar words used herein interchangeably mayrefer to, for example, one or more computers, computing entities,desktops, mobile phones, tablets, phablets, notebooks, laptops,distributed systems, kiosks, input terminals, servers or servernetworks, blades, gateways, switches, processing devices, processingentities, set-top boxes, relays, routers, network access points, basestations, the like, and/or any combination of devices or entitiesadapted to perform the functions, operations, and/or processes describedherein. External computing entities 102 can be operated by variousparties. As shown in FIG. 3, the external computing entity 102 caninclude an antenna 312, a transmitter 304 (e.g., radio), a receiver 306(e.g., radio), and a processing element 308 (e.g., CPLDs,microprocessors, multi-core processors, coprocessing entities, ASIPs,microcontrollers, and/or controllers) that provides signals to andreceives signals from the transmitter 304 and receiver 306,respectively.

The signals provided to and received from the transmitter 304 and thereceiver 306, respectively, may include signaling information/data inaccordance with air interface standards of applicable wireless systems.In this regard, the external computing entity 102 may be capable ofoperating with one or more air interface standards, communicationprotocols, modulation types, and access types. More particularly, theexternal computing entity 102 may operate in accordance with any of anumber of wireless communication standards and protocols, such as thosedescribed above with regard to the classification computing entity 106.In a particular embodiment, the external computing entity 102 mayoperate in accordance with multiple wireless communication standards andprotocols, such as UMTS, CDMA2000, 1×RTT, WCDMA, GSM, EDGE, TD-SCDMA,LTE, E-UTRAN, EVDO, HSPA, HSDPA, Wi-Fi, Wi-Fi Direct, WiMAX, UWB, IR,NFC, Bluetooth, USB, and/or the like. Similarly, the external computingentity 102 may operate in accordance with multiple wired communicationstandards and protocols, such as those described above with regard tothe classification computing entity 106 via a network interface 320.

Via these communication standards and protocols, the external computingentity 102 can communicate with various other entities using conceptssuch as Unstructured Supplementary Service Data (USSD), Short MessageService (SMS), Multimedia Messaging Service (MIMS), Dual-ToneMulti-Frequency Signaling (DTMF), and/or Subscriber Identity ModuleDialer (SIM dialer). The external computing entity 102 can also downloadchanges, add-ons, and updates, for instance, to its firmware, software(e.g., including executable instructions, applications, programmodules), and operating system.

According to one embodiment, the external computing entity 102 mayinclude location determining aspects, devices, modules, functionalities,and/or similar words used herein interchangeably. For example, theexternal computing entity 102 may include outdoor positioning aspects,such as a location module adapted to acquire, for example, latitude,longitude, altitude, geocode, course, direction, heading, speed,universal time (UTC), date, and/or various other information/data. Inone embodiment, the location module can acquire data, sometimes known asephemeris data, by identifying the number of satellites in view and therelative positions of those satellites (e.g., using global positioningsystems (GPS)). The satellites may be a variety of different satellites,including Low Earth Orbit (LEO) satellite systems, Department of Defense(DOD) satellite systems, the European Union Galileo positioning systems,the Chinese Compass navigation systems, Indian Regional Navigationalsatellite systems, and/or the like. This data can be collected using avariety of coordinate systems, such as the Decimal Degrees (DD);Degrees, Minutes, Seconds (DMS); Universal Transverse Mercator (UTM);Universal Polar Stereographic (UPS) coordinate systems; and/or the like.Alternatively, the location information/data can be determined bytriangulating the external computing entity's 102 position in connectionwith a variety of other systems, including cellular towers, Wi-Fi accesspoints, and/or the like. Similarly, the external computing entity 102may include indoor positioning aspects, such as a location moduleadapted to acquire, for example, latitude, longitude, altitude, geocode,course, direction, heading, speed, time, date, and/or various otherinformation/data. Some of the indoor systems may use various position orlocation technologies including RFID tags, indoor beacons ortransmitters, Wi-Fi access points, cellular towers, nearby computingdevices (e.g., smartphones, laptops) and/or the like. For instance, suchtechnologies may include the iBeacons, Gimbal proximity beacons,Bluetooth Low Energy (BLE) transmitters, NFC transmitters, and/or thelike. These indoor positioning aspects can be used in a variety ofsettings to determine the location of someone or something to withininches or centimeters.

The external computing entity 102 may also comprise a user interface(that can include a display 316 coupled to a processing element 308)and/or a user input interface (coupled to a processing element 308). Forexample, the user interface may be a user application, browser, userinterface, and/or similar words used herein interchangeably executing onand/or accessible via the external computing entity 102 to interact withand/or cause display of information/data from the classificationcomputing entity 106, as described herein. The user input interface cancomprise any of a number of devices or interfaces allowing the externalcomputing entity 102 to receive data, such as a keypad 318 (hard orsoft), a touch display, voice/speech or motion interfaces, or otherinput device. In embodiments including a keypad 318, the keypad 318 caninclude (or cause display of) the conventional numeric (0-9) and relatedkeys (#, *), and other keys used for operating the external computingentity 102 and may include a full set of alphabetic keys or set of keysthat may be activated to provide a full set of alphanumeric keys. Inaddition to providing input, the user input interface can be used, forexample, to activate or deactivate certain functions, such as screensavers and/or sleep modes.

The external computing entity 102 can also include volatile storage ormemory 322 and/or non-volatile storage or memory 324, which can beembedded and/or may be removable. For example, the non-volatile memorymay be ROM, PROM, EPROM, EEPROM, flash memory, MMCs, SD memory cards,Memory Sticks, CBRAM, PRAM, FeRAM, NVRAM, MRAM, RRAM, SONOS, FJG RAM,Millipede memory, racetrack memory, and/or the like. The volatile memorymay be RAM, DRAM, SRAM, FPM DRAM, EDO DRAM, SDRAM, DDR SDRAM, DDR2SDRAM, DDR3 SDRAM, RDRAM, TTRAM, T-RAM, Z-RAM, RIMM, DIMM, SIMM, VRAM,cache memory, register memory, and/or the like. The volatile andnon-volatile storage or memory can store databases, database instances,database management systems, data, applications, programs, programmodules, scripts, source code, object code, byte code, compiled code,interpreted code, machine code, executable instructions, and/or the liketo implement the functions of the external computing entity 102. Asindicated, this may include a user application that is resident on theentity or accessible through a browser or other user interface forcommunicating with the classification computing entity 106 and/orvarious other computing entities.

In another embodiment, the external computing entity 102 may include oneor more components or functionality that are the same or similar tothose of the classification computing entity 106, as described ingreater detail above. As will be recognized, these architectures anddescriptions are provided for exemplary purposes only and are notlimiting to the various embodiments.

In various embodiments, the external computing entity 102 may beembodied as an artificial intelligence (AI) computing entity, such as anAmazon Echo, Amazon Echo Dot, Amazon Show, Google Home, and/or the like.Accordingly, the external computing entity 102 may be configured toprovide and/or receive information/data from a user via an input/outputmechanism, such as a display, a camera, a speaker, a voice-activatedinput, and/or the like. In certain embodiments, an AI computing entitymay comprise one or more predefined and executable program algorithmsstored within an onboard memory storage module, and/or accessible over anetwork. In various embodiments, the AI computing entity may beconfigured to retrieve and/or execute one or more of the predefinedprogram algorithms upon the occurrence of a predefined trigger event.

IV. EXEMPLARY SYSTEM OPERATION

The operation of various embodiments of the present invention will nowbe described. As discussed herein, various embodiments are directed tosystems and methods for classification in hierarchical predictiondomains. Various embodiments of the disclosed techniques enableclassification in hierarchical prediction domains by using at least oneof online ML in hierarchical prediction domains, co-occurrence analysisin hierarchical prediction domains, fusion of structurally hierarchicalpredictions and structurally non-hierarchical predictions in inhierarchical prediction domains, fusion of structure-based predictionsand non-structure-based predictions in hierarchical prediction domains,and HPO predictions in the hierarchical HPO label domain.

A. Classification in a Hierarchical Prediction Domain

Various embodiments of the present invention are directed toclassification in a hierarchical prediction domain by using at least oneof structured input data and unstructured data. Structured data mayrefer to data that can be divided into semantically-defined data objectsbased on a predefined format of the data. Examples of structured datainclude data defined using a Structured Query Language (SQL), datadefined using a file format language (such as the JavaScript ObjectNotation (JSON) language, a Comma-Separated Value (CSV) language, or anExtensible Markup Language (XML) language), and/or the like. In thehealthcare context, structured data may include medical claims data,which may include information associated with each medical claim (e.g.,information about time of a medical operation associated with a medicalclaim, one or more operation codes associated with a medical claim, costof a medical operation associated with a medical, and/or the like.) in astructured format. Unstructured data may refer to data that cannot bedivided into semantically-defined data objects based on a predefinedformat of the data. Examples of unstructured data include various typesof natural language data, such as medical notes data which includesmedical notes provided by a medical provider. Although various solutionsexist for inferring semantically-defined data objects from unstructureddata (such as various natural language processing solutions), such aninference is distinct from a straightforward division of structured datainto semantically-defined data objects based on a predefined format ofthe structured data.

Thus, various embodiments of the present invention relate to performingpredictions related to prediction tasks characterized by ahierarchically complex prediction domain as well as a structurallycomplex input space. An example of a prediction task that present thecomplexities referred to herein is predicting HPO labels for a patientbased on medical data associated with the patient, such as medicalclaims data associated with the patient and medical notes dataassociated with the medical patient. The HPO label space, which providesa standardized vocabulary of phenotypic abnormalities associated withthousands of diseases, is an example of a hierarchical predictiondomain, as further described below. To perform HPO label predictionusing structured medical data and unstructured medical data, there is aneed for predictive analysis solutions that address the complexitiesassociated with the HPO label space as well as the complexitiesassociated with processing both structured medical data and unstructuredmedical data.

To perform predictions in a hierarchical prediction domains usingstructured input data and/or unstructured input data, variousembodiments of the present invention propose various arrangements of oneor more of the following ML models: an online ML model for processingstructured input data to generate structure-based predictions, aco-occurrence analysis ML model for processing structured input data togenerate structure-based predictions, a structured fusion ML model forcombining structure-based predictions, and an unstructured fusion MLmodel for combining structure-based predictions and non-structure-basedpredictions. In some embodiments, at least two of the mentioned MLmodels are organized in an ensemble architecture to generate a finalprediction based on predictions of the at least two ML models. In someembodiments, all of the mentioned ML models are organized in an ensemblearchitecture to generate a final prediction based on predictions of theat least two ML models.

FIGS. 4A and 4B depict two example ensemble architectures utilizing allof the four above-mentioned ML models. However, a person of ordinaryskill in the art will recognize that the four mentioned ML models can beutilized individually or in any particular combination of two or more ofthe four mentioned ML models. Furthermore, a person of ordinary skill inthe art will recognize that, if two or more ML models are utilized togenerate a prediction (e.g., all four mentioned ML models are utilizedto generate a prediction), the four ML models may be organized inaccordance with any ensemble architecture, including an ensemblearchitecture that is different from either or both of the ensemblearchitectures depicted in FIGS. 4A and 4B. Moreover, a person ofordinary skill in the art will recognize that one or more of each of thefour mentioned ML models may be utilized in combination of one or moreother ML models in accordance with various ensemble architectures togenerate a multi-model prediction framework. Thus, the depiction ofexample ensemble architectures in FIGS. 4A and 4B, and the accompanyingdescription of the noted example ensemble architectures provided herein,is not meant to be limiting as to the scope of the present invention.

FIG. 4A is an operational flow diagram for an ensemble architecture 410with an online ML model for processing structured input data, aco-occurrence analysis ML model for processing structured input data, astructured fusion ML model for combining structure-based predictions,and an unstructured fusion ML model for combining structure-basedpredictions and non-structure-based predictions, where the ensemblearchitecture 410 performs a structured fusion before performing anunstructured fusion. As depicted in the ensemble architecture 410, theonline learning unit 111 retrieves the structured input data 121 fromthe storage subsystem and processes the structured input data 121 inaccordance with an online ML model to generate one or more onlinelearning predictions. Moreover, the co-occurrence analysis unit 112retrieves the structured input data 121 from the storage subsystem andprocesses the structured input data 121 in accordance with aco-occurrence analysis ML model to generate one or more co-occurrenceanalysis predictions. In some embodiments, both the one or more onlinelearning predictions and the one or more co-occurrence analysispredictions are structure-based predictions, i.e., predictions generatedbased on structured input data.

In some embodiments, to generate the one or more online learningpredictions, the online learning unit 111 processes the structured inputdata 121 in accordance with an online ML model. The online ML model maybe a ML model that utilizes sequential updates to a ML model in order toinfer a relationship between a prediction input space and a predictionoutput space. For example, the online ML model may be an online ML modelthat utilizes structured medical claim data to generate one or more HPOlabel predictions. An example of such an online ML model may be a MLmodel that utilizes an FTRL ML algorithm. In some embodiments, the oneor more online learning predictions may be structurally hierarchicalpredictions. A structurally hierarchical prediction may be a predictiondetermined based at least in part on a position of a correspondingprediction node in a structural hierarchy characterizing thehierarchical prediction domain that includes the correspondingprediction node. For example, in some embodiments, an online ML modelmay be configured to generate the one or more online learningpredictions based on prediction values for prediction nodes deemed to bemost-dependent prediction nodes in a structurally hierarchycharacterizing the prediction domain associated with the online MLmodel. In at least some of those embodiments, the online ML model willnot generate predictions that correspond to non-most-dependent nodes inthe structural hierarchically characterizing the prediction domainassociated with the online ML model.

In some embodiments, to generate the one or more co-occurrence analysispredictions, the co-occurrence analysis unit 112 processes thestructured input data 121 in accordance with a co-occurrence analysis MLmodel. The co-occurrence analysis ML model may be a ML model that infersone or more statistical relationships based on co-occurrences ofparticular prediction input values (e.g., medical codes in medicalclaims data, such as medical codes corresponding to the 10th Revision ofthe International Classification of Diseases Procedure ClassificationSystem (ICD-10-PCS) and/or medical codes corresponding to the 10thRevision of the International Classification of Diseases ClinicalModification (ICD-10-CM) system) and particular prediction output values(e.g., HPO labels). In some embodiments, the one or more co-occurrenceanalysis predictions may be structurally non-hierarchical predictions. Astructurally non-hierarchical prediction may be a prediction determinedwithout regard to a position of the corresponding prediction node in astructural hierarchy characterizing the hierarchical prediction domainthat includes the corresponding prediction node. For example, in someembodiments, a co-occurrence analysis ML model may be configured togenerate the one or more co-occurrence analysis predictions based onprediction values for prediction nodes regardless of whether theprediction nodes are deemed to be dependent prediction nodes in astructurally hierarchy characterizing the prediction domain associatedwith the co-occurrence analysis ML model. In at least some of thoseembodiments, the co-occurrence analysis ML model may generatepredictions that correspond to both most-dependent nodes andnon-most-dependent nodes in the structural hierarchically characterizingthe prediction domain associated with the online ML model.

Returning to FIG. 4A, the ensemble architecture 410 further includesprocessing the online learning predictions generated by the onlinelearning unit 111 and the co-occurrence analysis predictions generatedby the co-occurrence analysis unit 112 using the structured fusion unit113 to generate one or more structured fusion predictions. To processthe online learning predictions generated by the online learning unit111 and the co-occurrence analysis predictions generated by theco-occurrence analysis unit 112 to generate one or more structuredfusion predictions, the structured fusion unit 113 may utilize astructured fusion ML model. The structured fusion ML model may be a MLmodel configured to generate/predict one or more structured fusionpredictions based on unfused predictions generated by two or moreunfused ML models, where the unfused predictions include at least oneprediction generated by a ML model configured to generate/predictstructurally hierarchical predictions and at least one predictiongenerated by a ML model configured to generate/predict structurallynon-hierarchical predictions. In the particular ensemble architecture410 depicted in FIG. 4A, the structured fusion ML model utilized by thestructured fusion unit 113 is configured to generate one or morestructured fusion predictions based on unfused predictions generated bytwo or more unfused ML models: structurally hierarchical predictions(i.e., the one or more online learning predictions) generated by theonline ML model utilized by the online learning unit 111 andstructurally non-hierarchical predictions (i.e., the one or moreco-occurrence analysis predictions) generated by the co-occurrenceanalysis ML model utilized by the co-occurrence analysis unit 112. Whilethe particular ensemble architecture 410 depicted in FIG. 4A includes astructured fusion unit 113 configured to process structure-basedpredictions (i.e., the one or more online learning predictions and theone or more co-occurrence analysis predictions, both of which aredetermined based on the structured input data 121), a person of ordinaryskill in the art will recognize that the structured fusion ML modelutilized by the structured fusion unit 113 may be configured to processstructurally hierarchical predictions determined based on unstructuredinput data and/or structurally non-hierarchical predictions determinedbased on unstructured input data.

Returning to FIG. 4A, the ensemble architecture 410 further includesprocessing the one or more structured fusion predictions generated bythe structured fusion unit 113 with one or more non-structure-basedpredictions determined based on the unstructured input data 122 usingthe unstructured fusion unit 114 to generate one or more finalpredictions 401. To process the one or more structured-fusionpredictions generated by the structured fusion unit 113 with one or morenon-structure-based predictions determined based on the unstructuredinput data 122 in order to generate one or more final predictions 401,the unstructured fusion unit 114 may utilize an unstructured fusion MLmodel. The unstructured fusion ML model may be a ML model configured togenerate/predict one or more unstructured-fused predictions based onunfused predictions generated by two or more unfused ML models, wherethe unfused predictions include at least one prediction generated by aML model configured to generate/predict structure-based predictions andat least one prediction generated by a ML model configured togenerate/predict non-structure-based predictions. In some embodiments, astructure-based prediction is a prediction determined based onstructured input data, while a non-structure-based prediction is aprediction determined based on unstructured input data.

In the particular ensemble architecture 410 depicted in FIG. 4A, theunstructured fusion ML model utilized by the unstructured fusion unit114 is configured to generate unstructured-fused predictions based onstructured-fused predictions generated by the structured fusion unit 113and non-structure-based predictions generated based on the unstructuredinput data 122 stored in the storage subsystem 108. However, a person ofordinary skill in the art will recognize that the unstructured fusion MLmodel utilized by the unstructured fusion unit 114 may be configured toprocess any combination of structure-based predictions andnon-structure-based predictions. In some embodiments, the unstructuredfusion unit 114 is further configured to perform natural languageprocessing on the unstructured input data 122 to generate unfusednon-structure-based predictions based on the unstructured input data122, while in other embodiments the unstructured input data 122 includespre-determined unfused non-structure-based predictions determined basedon particular natural language input data (e.g., feature data extractedfrom medical notes data using a synonym-based natural languageprocessing analysis).

FIG. 4B is an operational flow diagram for an ensemble architecture 450with an online ML model for processing structured input data, aco-occurrence analysis ML model for processing structured input data, astructured fusion ML model for combining structure-based predictions,and an unstructured fusion ML model for combining structure-basedpredictions and non-structure-based predictions, where the ensemblearchitecture 410 performs an unstructured fusion before performing astructured fusion. As depicted in the ensemble architecture 450, theco-occurrence analysis unit 112 retrieves the structured input data 121from the storage subsystem and processes the structured input data 121in accordance with a co-occurrence analysis ML model to generate one ormore co-occurrence analysis predictions.

In some embodiments, to generate the one or more co-occurrence analysispredictions, the co-occurrence analysis unit 112 processes thestructured input data 121 in accordance with a co-occurrence analysis MLmodel. The co-occurrence analysis ML model may be a ML model that infersone or more statistical relationships based on co-occurrences ofparticular prediction input values (e.g., medical codes in medicalclaims data, such as medical codes corresponding to the ICD-10-PCSand/or ICD-10-CM systems) and particular prediction output values (e.g.,HPO labels). In some embodiments, the one or more co-occurrence analysispredictions may be structurally non-hierarchical predictions. Forexample, in some embodiments, a co-occurrence analysis ML model may beconfigured to generate the one or more co-occurrence analysispredictions based on prediction values for prediction nodes regardlessof whether the prediction nodes are deemed to be dependent predictionnodes in a structurally hierarchy characterizing the prediction domainassociated with the co-occurrence analysis ML model.

Returning to FIG. 4B, the ensemble architecture 450 further includesprocessing the one or more co-occurrence analysis predictions generatedby the co-occurrence analysis unit 112 and one or morenon-structure-based predictions generated based on the unstructuredinput data 122 using the unstructured fusion unit 114 to generate one ormore unstructured-fused predictions. To process the one or moreco-occurrence analysis predictions generated by the co-occurrenceanalysis unit 112 and the one or more non-structure-based predictionsgenerated based on the unstructured input data 122 in order to generateone or more unstructured-fused predictions, the unstructured fusion unit114 may utilize an unstructured fusion ML model. The unstructured fusionML model may be a ML model configured to generate/predict one or moreunstructured-fused predictions based on unfused predictions generated bytwo or more unfused ML models, where the unfused predictions include atleast one prediction generated by a ML model configured togenerate/predict structure-based predictions and at least one predictiongenerated by a ML model configured to generate/predictnon-structure-based predictions.

In the particular ensemble architecture 450 depicted in FIG. 4B, theunstructured fusion ML model utilized by the unstructured fusion unit114 is configured to generate unstructured-fused predictions based onstructure-based unfused predictions generated by the co-occurrenceanalysis unit 112 and non-structure-based predictions generated based onthe unstructured input data 122 stored in the storage subsystem 108.However, a person of ordinary skill in the art will recognize that theunstructured fusion ML model utilized by the unstructured fusion unit114 may be configured to process any combination of structure-basedpredictions and non-structure-based predictions. In some embodiments,the unstructured fusion unit 114 is further configured to performnatural language processing on the unstructured input data 122 togenerate unfused non-structure-based predictions based on theunstructured input data 122, while in other embodiments the unstructuredinput data 122 include pre-determined unfused non-structure-basedpredictions determined based on particular natural language input data(e.g., feature data extracted from medical notes data using asynonym-based natural language processing analysis).

Returning to FIG. 4B, the ensemble architecture 450 further includesretrieving the structured input data 121 from the storage subsystem andprocessing the structured input data 121 using the online learning unit111 to generate one or more online learning predictions. To process thestructured input data 121 to generate one or more online learningpredictions, the online learning unit 111 may utilize an online MLmodel. The online ML model may be a ML model that utilizes sequentialupdates to a ML model in order to infer a relationship between aprediction input space and a prediction output space. For example, theonline ML model may be an online ML model that utilizes structuredmedical claim data to generate one or more HPO label predictions, suchas a ML model that utilizes a FTRL ML algorithm. In some embodiments,the one or more online learning predictions may be structurallyhierarchical predictions. For example, in some embodiments, an online MLmodel may be configured to generate the one or more online learningpredictions based on prediction values for prediction nodes deemed to bechildren prediction nodes in a structurally hierarchy characterizing theprediction domain associated with the online ML model.

The ensemble architecture 450 further includes processing theunstructured-fused predictions generated by the unstructured fusion unit114 and the online learning predictions generated by the online learningunit 111 using the structured fusion unit 113 to generate the finalpredictions 401. In some embodiments, to process the unstructured-fusedpredictions generated by the unstructured fusion unit 114 and the onlinelearning predictions generated by the online learning unit 111 in orderto generate the final predictions 401, the structured fusion unit 113utilizes a structured fusion ML model. The structured fusion ML modelmay be a ML model configured to generate/predict one or morestructured-fused predictions based on unfused predictions generated bytwo or more unfused ML models, where the unfused predictions include atleast one prediction generated by a ML model configured togenerate/predict structurally hierarchical predictions and at least oneprediction generated by a ML model configured to generate/predictstructurally non-hierarchical predictions.

In the particular ensemble architecture 410 depicted in FIG. 4B, thestructured fusion ML model utilized by the structured fusion unit 113 isconfigured to generate one or more structure-based predictions based onunfused predictions generated by two or more unfused ML models:structurally hierarchical predictions (i.e., the one or more onlinelearning predictions) generated by the online ML model utilized by theonline learning unit 111 and structurally non-hierarchical predictions(i.e., the one or more unstructured-fused predictions) generated by theunstructured fusion ML model utilized by the co-unstructured fusion unit114. While the particular ensemble architecture 410 depicted in FIG. 4Aincludes a structured fusion unit 113 configured to process bothstructure-based predictions (i.e., the one or more online learningpredictions) and non-structure-based predictions (the one or moreunstructured-fused predictions), a person of ordinary skill in the artwill recognize that the structured fusion ML model utilized by thestructured fusion unit 113 may be configured to process any combinationof structurally hierarchical predictions (e.g., structurallyhierarchical predictions generated based on structured data,unstructured, and/or a fusion of structure-based predictions andnon-structure-based predictions) and structurally non-hierarchicalpredictions (e.g., structurally non-hierarchical predictions generatedbased on structured data, unstructured, and/or a fusion ofstructure-based predictions and non-structure-based predictions).

B. Online ML in a Hierarchical Prediction domain

Online learning is a method of ML in which a ML model is sequentiallyupdated over time based on incoming training data. During training, someonline learning algorithms aim to set parameters of a predictionfunction in a manner that optimizes co-occurrence of the predictionfunction with existing training data, including new training data itemsand/or sequentially updated training data items. For example, onlinelearning algorithms are typically utilized to generate recommendationsfor a user (e.g., promotional recommendations for a user), where theuser reaction to the recommendation is in turn utilized to update a MLmodel. In some online learning algorithms, a positive user reaction(e.g., a selection of a link corresponding to a recommendation) is usedto change model parameters in a manner that increases a likelihood offuture generation of the recommendation and decreases a likelihood offuture generation of other recommendations, while a negative userreaction (e.g., lack of selection of a link corresponding to arecommendation) is used to change model parameters in a manner thatdecreases a likelihood of future generation of the recommendation andincreases a likelihood of future generation of other recommendations.

Hierarchical prediction domains present unique challenges for onlinelearning algorithms. When utilized to generate predictions related tohierarchical prediction domains, online learning algorithms shouldaccommodate hierarchical predictive relationships between variousprediction nodes in determining how to interpret incoming training data.Without applying appropriate operational adjustments that addresshierarchical nature of a relevant prediction domain, online learningalgorithms will require higher amounts of training data, will takelonger to train, and will once trained be less accurate and reliable.Because of those challenges, various existing online learning algorithmsare ill-suited for efficiently and reliably performing classification inrelation to hierarchical prediction domains.

Various embodiments of the present invention address efficiency andreliability challenges related to utilizing online learning algorithmsto generate predictions related to hierarchical prediction domains.According to one aspect that relates to improving efficiency andreliability of online learning in hierarchical prediction domains,various embodiments of the present invention eliminate a bias term usedto penalize lack of selection of a prediction node, as hierarchicalpredictive relationships complicate implications of such a lack ofselection for adjusting model parameters. For example, selection of aprediction node may have different implications for prediction nodesthat are dependent on the particular prediction node, prediction nodesfrom which the particular prediction node depends, and other predictionnodes without hierarchical relationships with the particular predictionnode. To address such complications, various embodiments of the presentinvention will not penalize lack of selection of a particular node whenadjusting parameters of a relevant ML model. In doing so, variousembodiments of the present invention address efficiency and reliabilitychallenges related to utilizing online learning algorithms to generatepredictions related to hierarchical prediction domains.

According to another aspect that relates to improving efficiency andreliability of online learning in hierarchical prediction domains,various embodiments of the present invention perform predictiveinferences by selecting prediction nodes having sufficiently highpredictive scores starting from dependent prediction nodes. In doing so,the mentioned embodiments of the present invention increase thelikelihood that prediction nodes having more detailed semanticimplications (e.g., more “meaningful” prediction nodes) will be selectedover prediction nodes having less detailed semantic associations, thusincreasing the reliability of the predictive analysis performed usingonline learning. For example, a prediction node associated with athoracolumbar scoliosis HPO label will have a higher chance of selectionthat a prediction node associated with a scoliosis HPO label, as theformer has a more meaningful semantic association than the former. Thiswill lead to generation of structurally hierarchical predictions whichhave greater predictive utility. Moreover, selection of predictionlabels in a hierarchical manner decreases the range of predictive scoresthat need to be analyzed during a predictive inference. This is because,according to various embodiments of the present invention, predictiveinference will halt if a requisite number of prediction nodes areselected inmost-dependent nodes. Such techniques have the addedadvantage of increasing efficiency of online learning in hierarchicalprediction domains by decreasing the range of prediction nodes whichneed to be traversed before a final prediction output is generated aspart of a particular predictive inference. Thus, by selecting predictionnodes having sufficiently high predictive scores starting from dependentprediction nodes, various embodiments of the present invention addressefficiency and reliability challenges related to utilizing onlinelearning algorithms to generate predictions related to hierarchicalprediction domains.

According to yet another aspect that relates to improving efficiency andreliability of online learning in hierarchical prediction domains,various embodiments of the present invention provide techniques forefficiently storing and retrieving training data. Online learningalgorithms face challenges related to efficiently storing and retrievingtraining data during training of relevant ML models. In the absence ofefficient solutions for storing and retrieving training data duringtraining of relevant ML models, many conventional online learningalgorithms are slow to train, which undermines the utility of suchalgorithms for predictive tasks that require real-time training and/orreal-time model updates. To address such challenges, various embodimentsof the present invention store training data entries in a highly sparsevector using a hashing mechanism. This aspect serves to increaseefficiency and reliability of online learning in all domains, includingin in hierarchical prediction domains. Thus, by storing training dataentries in a highly sparse vector using a hashing mechanism, variousembodiments of the present invention address efficiency and reliabilitychallenges related to utilizing online learning algorithms to generatepredictions related to hierarchical prediction domains.

FIG. 5 provides a flowchart diagram of an example process 500 fortraining an online ML model to perform predictive inferences related toa hierarchical prediction domain. Via the various steps/operations ofprocess 500, the model generation unit 115 of the classificationcomputing entity 106 can train predictive inferences using a ML modelthat eliminates a bias term used to penalize lack of selection of aprediction node, selects prediction nodes having sufficiently highpredictive scores starting from dependent prediction nodes, and/orstores training data entries in a highly sparse vector using a hashingmechanism.

The process 500 begins at step/operation 501 when the model generationunit 115 prepares hierarchically-expanded training data. In someembodiments, to prepare hierarchically-expanded training data, the modelgeneration unit 115 first retrieves raw training data 124 from thestorage subsystem 108. The raw training data 124 may include, for eachtraining entity (e.g., patient), one or more training features (e.g.,medical codes) and one or more training prediction labels (e.g., one ormore HPO labels), where at least one of the training prediction labelsis associated with a hierarchical predictive relationship. Afterretrieving raw training data 124, the model generation unit 115 mayidentify each prediction node that is a parent to the at least one ofthe training prediction labels and generate a corresponding trainingdata object for each identified prediction node. For example, given rawtraining data objects {A1, A2, A3→X1}, {A4, A5, A6→X2}, and {A7, A8,A9→X3} (where {α, β, ç→{acute over (ε)}} denotes that training featuresα, β, ç are associated with the training prediction label {acute over(ε)}), and further given that X1 and X2 both depend from X3, the modelgeneration unit 115 may generate the following hierarchically-expandedtraining data objects: {A1, A2, A3→X1}, {A4, A5, A6→X2}, {A7, A8,A9→X3}, {A1, A2, A3→X3}, {A4, A5, A6→X3}.

In some embodiments, step/operation 501 can be performed in accordancewith the various steps/operations depicted in FIG. 6. The processdepicted in FIG. 6 begins at step/operation 601 when the modelgeneration unit 115 obtains one or more training data objects eachassociated with one or more training features and one or more trainingprediction labels. In some embodiments, the one or more trainingprediction labels are ground-truth predictions associated withparticular training features. For example, training features may includemedical procedure codes associated with medical procedures for apatient, while training prediction labels may include HPO labels for thepatient.

At step/operation 602, the model generation unit 115 converts each setof training features for a training data object obtained instep/operation 601 into a feature string for the training data object.In some embodiments, the model generation unit 115 generates a featurestring for each training data object identified in step/operation 601based on each training feature associated with the training data object.For example, for training data object {A1, A2, A3→X1} (where {α, β,ç→{acute over (ε)}} denotes that training features α, β, ç areassociated with the training prediction label {acute over (ε)}), themodel generation unit 115 may generate the following feature string forthe respective training data object: A1A2A3X1. In some embodiments, themodel generation unit 115 generates one or more inferred features fromthe training features identified in step/operation 601. Then, for eachtraining data object identified in step/operation 601, the modelgeneration unit 115 generates a feature string for each training dataobject identified in step/operation 601 based on each inferred trainingfeature associated with the training data object. For example, fortraining data object {A1, A2, A3→X1} (where {α, β, ç→{acute over (ε)}}denotes that training features α, β, ε are associated with the trainingprediction label {acute over (ε)}), the model generation unit 115 maygenerate the inferred training features B1 and B2 based on the trainingfeatures A1, A2, and A3 and then generate the following feature stringfor the respective training data object: B1B2.

At step/operation 603, the model generation unit 115 identifies, foreach training data object obtained in step/operation 601, each parentprediction label for at least one prediction label associated with thetraining data object, where the parent prediction label for a particularprediction label is a prediction label from which the particularprediction label depends according to the hierarchical predictiverelationships characterizing a relevant hierarchical prediction domain.For example, given raw training data object {A1, A2, A3→X1} (where {α,β, ç→{acute over (ε)}} denotes that training features α, β, ç areassociated with the training prediction label {acute over (ε)}), andfurther given that X1 depends from the prediction label X3, the modelgeneration unit 115 may identify the prediction label X3 as a parentprediction label for a prediction label associated with the mentionedtraining data object. In some embodiments, to identify parent predictionlabels, the model generation unit 115 utilizes a graph traversalalgorithm.

At step/operation 604, the model generation unit 115 generateshierarchically-expanded training data based on each feature stringgenerated in step/operation 602, each training prediction label obtainedin step/operation 601, and each parent prediction label identified instep/operation 603. In some embodiments, to generate thehierarchically-expanded training data, the model generation unit 115generates a hierarchically-expanded training data object for each parentprediction node which associates the parent prediction node to thetraining features that are associated with the prediction node whichdepends from the parent prediction node. For example, given raw trainingdata objects {A1, A2, A3→X1}, {A4, A5, A6→X2}, and {A7, A8, A9→X3 }(where {α, β, ç→{acute over (ε)}} denotes that training features α, β, çare associated with the training prediction label {acute over (ε)}), andfurther given that X1 and X2 both depend from X3, the model generationunit 115 may generate the following hierarchically-expanded trainingdata objects, which include the latter hierarchically-expanded trainingdata objects for the parent prediction node X3: {A1, A2, A3→X1}, {A4,A5, A6→X2}, {A7, A8, A9→X3}, {A1, A2, A3→X3}, {A4, A5, A6→X3}.

FIGS. 7-9 provide operational examples of various aspects of an exampleprocess for generating hierarchically-expanded training data objectsbased on particular raw training data objects. In particular, FIG. 7provides an operational example of a raw training data object set 700that includes three training data objects: a first training data object701 which associates a particular training feature A to a particulartraining prediction label related to abnormality of the curvature of thevertebral column; a second training data object 702 which associates aparticular training feature B to a particular training prediction labelrelated to abnormality of the thoracic spine; and a third training dataobject 703 which associates a particular training feature C to aparticular prediction label related to kyphosis. The three training dataobjects in the raw training data object set 700 may be represented as{A→abnormality of the curvature of the vertebral column} for the firsttraining data object 701, {B→abnormality of the thoracic spine} for thesecond training data object 702, and {C→kyphosis} for the third trainingdata object 703, where {α, β, ç→{acute over (ε)}} denotes that trainingfeatures α, β, ç are associated with the training prediction label{acute over (ε)}. A feature as depicted in FIG. 7 may include one ormore feature attributes, e.g., one or more patient descriptorattributes, one or more procedure attributes, and/or one or more ICDcodes, etc.

FIG. 8 provides an operational example of a hierarchical predictiondomain 800. The depicted exemplary hierarchical prediction domain 800comprises a number of prediction nodes each of which has a number ofpredictive hierarchical relationships, including child-hierarchicalrelationships and/or parent-dependence relationship. For example,prediction node 804 associated with the abnormality of the thoracicspine (which relates to the training feature for the second trainingdata object 702 in the raw training object data set 700 of FIG. 7) haschild-dependence relationships with the following prediction nodes:prediction node 803 associated with thoracic kyphosis, prediction node802 associated with thoracolumbar kyphosis, and prediction node 801associated with thoracolumbar kyphoscoliosis. As another example,prediction node 804 has parent-dependence relationships with thefollowing prediction nodes: prediction node 807 associated with theabnormality of the thorax, prediction node 808 associated with theabnormality of the vertebral column, prediction node 809 associated withabnormal axial skeleton morphology, prediction node 810 associated withthe abnormality of the skeletal morphology, prediction node 811associated with abnormality of the skeletal system, and prediction node812 associated with the phenotypic abnormality.

As yet another example in the depicted exemplary hierarchical predictiondomain 800 of FIG. 8, prediction node 806 associated with theabnormality of the curvature of the vertebral column (which relates tothe training feature for the first training data object 701 in the rawtraining object data set 700 of FIG. 7) is associated with the followingparent-dependence relationships: prediction node 808 associated with theabnormality of the vertebral column, prediction node 809 associated withabnormal axial skeleton morphology, prediction node 810 associated withthe abnormality of the skeletal morphology, prediction node 811associated with abnormality of the skeletal system, and prediction node812 associated with the phenotypic abnormality. As a further example,prediction node 805 associated with kyphosis (which relates to thetraining feature for the third training data object 703 in the rawtraining object data set 700 of FIG. 7) is associated with the followingparent-dependence relationships: prediction node 806 associated with theabnormality of the curvature of the vertebral column (which relates tothe training feature for the first training data object 701 in the rawtraining object data set 700 of FIG. 7), prediction node 808 associatedwith the abnormality of the vertebral column, prediction node 809associated with abnormal axial skeleton morphology, prediction node 810associated with the abnormality of the skeletal morphology, predictionnode 811 associated with abnormality of the skeletal system, andprediction node 812 associated with the phenotypic abnormality.

FIG. 9 provides an operational example of a hierarchically-expandedtraining data object set 900 which includes, for each raw training dataobject depicted in the raw training object data set 700 of FIG. 7, ahierarchically-expanded training data object with the parent predictionlabels for the training prediction label in the raw training data objectin addition to the training prediction label itself. Thus, thehierarchically-expanded training data object set 900 includes a firsthierarchically-expanded training data object 901 which includes, forfeature A associated with the first training data object 701 in the rawtraining object data set 700, the following prediction labels:abnormality of the curvature of the vertebral column (corresponding toprediction node 806), abnormality of the vertebral column (whichcorresponds to prediction node 808), abnormal axial skeleton morphology(which corresponds to prediction node 809), skeletal morphology (whichcorresponds to prediction node 810), abnormality of the skeletal system(which corresponds to prediction node 811), and phenotypic abnormality(which corresponds to prediction node 812). Furthermore, thehierarchically-expanded training data object set 900 includes a secondhierarchically-expanded training data object 902 which includes, forfeature B associated with the second training data object 702 in the rawtraining object data set 700, the following prediction labels:abnormality of the thoracic spine (which corresponds to prediction node804), abnormality of the thorax (which corresponds to prediction node807), abnormality of the vertebral column (which corresponds toprediction node 808), abnormal axial skeleton morphology (whichcorresponds to prediction node 809), skeletal morphology (whichcorresponds to prediction node 810), abnormality of the skeletal system(which corresponds to prediction node 811), and phenotypic abnormality(which corresponds to prediction node 812). Moreover, thehierarchically-expanded training data object set 900 includes a thirdhierarchically-expanded training data object 903 which includes, forfeature B associated with the third training data object 703 in the rawtraining object data set 700, the following prediction labels: kyphosis(which corresponds to prediction node 805), abnormality of the curvatureof the vertebral column (corresponding to prediction node 806),abnormality of the vertebral column (which corresponds to predictionnode 808), abnormal axial skeleton morphology (which corresponds toprediction node 809), skeletal morphology (which corresponds toprediction node 810), abnormality of the skeletal system (whichcorresponds to prediction node 811), and phenotypic abnormality (whichcorresponds to prediction node 812).

In some embodiments, the model generation unit 115 may generate thefollowing hierarchically-expanded training data objects based on thehierarchically-expanded training data object set 900 depicted in FIG. 9(where {α, β, ç→{acute over (ε)}} denotes that training features α, β, çare associated with the training prediction label {acute over (ε)}):{Feature A→Abnormality of the Curvature of the Vertebral Column},{Feature A→Abnormality of the Vertebral Column}, {Feature A→Abnormalityof the Axial Skeleton Morphology}, {Feature A→Abnormality of theSkeletal System}, {Feature A→Phenotypic Abnormality}, {FeatureB→Abnormality of the Thoracic Spine}, {Feature B→Abnormality of theThorax}, {Feature B→Abnormality of the Vertebral Column}, {FeatureB→Abnormality of the Axial Skeleton Morphology}, {Feature B→Abnormalityof the Skeletal System}, {Feature B→Phenotypic Abnormality}, {Feature C,Kyphosis}, {Feature C→Abnormality of the Vertebral Column}, {FeatureC→Abnormality of the Axial Skeleton Morphology}, {Feature C→Abnormalityof the Skeletal System}, {Feature C→Phenotypic Abnormality}.

Returning to FIG. 5, at step/operation 502, the model generation unit115 initializes an FTRL ML model. The FTRL ML model may be an example ofan online learning algorithm which is configured to perform at least thefollowing operations: (1) obtain a prediction input (e.g., an inputvector); (2) obtain a parameter data object (e.g., a parameter dataobject) having an initial value; (3) perform a prediction data object(e.g., a prediction vector) using the prediction object and theparameter data object (e.g., using logistic regression with a sigmoidfunction); (4) obtain a ground-truth observation data object (e.g., aground-truth observation data object); and (5) update the parameter dataobject based on the ground-truth observation data object (e.g., based ona gradient of the error between the prediction data object and theground-truth observation data object, for example by using onlinegradient descent). In many conventional FTRL ML models, the initialvalue of the parameter data object is configured to generate azero-valued prediction data object and the update in the fifth operationis characterized by a reward term configured to reward occurrence of apositive ground-truth observation data object (e.g., a ground-truthobservation data object indicating selection of a predicted item by anend-user) and a bias term configured to penalize lack of occurrence of anegative ground-truth observation data object (e.g., a ground-truthobservation data object indicating lack of selection of a predicted itemby an end-user).

In some embodiments, step/operation 502 may be performed in accordancewith the various steps/operations depicted in FIG. 10, which depicts thesteps/operations of an example process for an generating initial weightvalues for an FTRL ML model. The example process depicted in FIG. 10begins at step/operation 1001 when the model generation unit 115 obtainsan FTRL ML model characterized by an initial value and a bias term. Insome embodiments, the FTRL ML model in step/operation 1001 integrates atleast some of the steps/operations denoted in the below algorithm:

Algorithm 1 Per-Coordinate FTRL-Proximal with L₁ and L₂ Regularizationfor Logistic Regression #With per-coordinate learning rates of Eq. (2).Input: parameters α, β, λ₁, λ₂ (∀i ∈ {1, . . . , d}), initialize z_(i) =0 and n_(i) = 0 for t = 1 to T do  Receive feature vector x_(t) and letI = {i | x_(i) ≠ 0}  For i ∈ I compute  $w_{t,i} = \{ \begin{matrix}0 & {{{if}\mspace{14mu} {z_{i}}} \leq \lambda_{1}} \\{{- ( {\frac{\beta + \sqrt{n_{i}}}{\alpha} + \lambda_{2}} )^{- 1}}( {z_{i} - {{{sgn}( z_{i} )}\lambda_{1}}} )} & {{otherwise}.}\end{matrix} $  Predict p_(t) = σ(x_(t) · w) using the w_(t,i)computed above  Observe label y_(t) ∈ {0, 1}  for all i ∈ I do   g_(i) =(p_(t) − y_(t))x_(i) #gradient of loss w.r.t. w_(i)   $\sigma_{i} = {{\frac{1}{\alpha}( {\sqrt{n_{i} + g_{i}^{2}} - \sqrt{n_{i}}} )\mspace{31mu} \# {equals}\mspace{14mu} \frac{1}{\eta_{t,i}}} - \frac{1}{\eta_{{t - 1},i}}}$  z_(i) ← z_(i) + g_(i) − σ_(i)w_(t,i)   n_(i) ← n_(i) + g_(i) ²  endfor end for Algorithm 1 (from McMahan et al., “Ad-Click Prediction: AView from the Trenches,” (2015), available online athttp://www.eecs.tufts.edu/~dsculley/papers/ad-click-prediction.pdf)

At step/operation 1002, the model generation unit 115 sets the initialvalue of the FTRL ML model (e.g., the values corresponding to z_(i) inthe above-depicted algorithm) to a negative value. In some embodiments,the negative value is determined based on an expected distribution ofthe ratio of positive ground-truth observations to negative-ground-truthobservations among all the training data objects. At step/operation1003, the model generation unit 115 removes the bias term from the FTRLML model.

Returning to FIG. 5, at step/operation 503, the model generation unit115 generates training data objects based on the hierarchically-expandedtraining data objects generated in step/operation 501. In someembodiments, to generate training data objects based on thehierarchically-expanded training data objects generated instep/operation 501, the model generation unit 115 associates eachhierarchically-expanded data object generated in step/operation 604 to aprediction label for the hierarchically-expanded data object. Forexample, given hierarchically-expanded training data objects {A1, A2,A3→X1}, {A4, A5, A6→X2}, {A7, A8, A9→X3}, {A1, A2, A3→X3}, {A4, A5,A6→X3} where {α, β, ç→{acute over (ε)}} denotes that training featuresα, β, ç are associated with the training prediction label {acute over(ε)}, the model generation unit 115 may associate the firsthierarchically-expanded training data object to the prediction label X1,the second hierarchically-expanded training data object to theprediction label X2, and the latter three hierarchically-expandedtraining data objects to the prediction label X3. The model generationunit 115 may thereafter generate the training data objects based on thenoted associations.

At step/operation 504, the model generation unit 115 generates appendedtraining data objects based on the training data objects generated instep/operation 503. In some embodiments, to generate the appendedtraining data objects based on the training data objects generated instep/operation 503, the model generation unit 115 appends each trainingdata object generated in step/operation 503 to the prediction labeldetermined to be associated with the corresponding training data object.For example, given the training data objects {A1, A2, A3→X1} associatedwith the prediction label X1, {A4, A5, A6→X2} associated with theprediction label X2, {A7, A8, A9→X3} associated with the predictionlabel X3, {A1, A2, A3→X3} associated with the prediction label X3,and{A4, A5, A6→X3} associated with the prediction label X3, where {α, β,ç→{acute over (ε)}} denotes that training features α, β, ç areassociated with the training prediction label {acute over (ε)}, themodel generation unit 115 may generate the following appended trainingdata objects: A1A2A3X1, A4A5A6X2, A7A8A9X3, A1A2A3X3, and A4A5A6X3.

In some embodiments, step/operation 504 may be performed in accordancewith the various steps/operations depicted in FIG. 11, which depicts thesteps/operations of an example process for storing appended trainingdata objects. The example process depicted in FIG. 11 begins atstep/operation 1101 when the model generation unit 115 appends eachfeature string generated in step/operation 602 to each correspondingprediction label for the feature string and to each parent predictionlabel for the feature string identified in step/operation 603 togenerate one or more appended training data objects for the featurestring. For example, given the training feature string A1A2A3 associatedwith the prediction node X1 which is in turn a hierarchical child of theprediction node X3, the model generation unit 115 may generate thefollowing appended training data objects for the feature string A1A2A3:A1A2A3X1 and A1A2A3X3. As another example, given the training featurestring A4A5A6 associated with the prediction node X2 which is in turn ahierarchical child of the prediction node X3, the model generation unit115 may generate the following appended training data objects for thefeature string A4A5A6: A4A5A6X2 and A4A5A6X3. As a further example,given the training feature string A7A8A9X3 associated with theprediction node X3 which is in turn a hierarchical child of theprediction node X4, the model generation unit 115 may generate thefollowing appended training data objects for the feature stringA7A8A9X3: A7A8A9X3 and A7A8A9X4.

At step/operation 1102, the model generation unit 115 generates acorresponding encoding location for each appended training data entrygenerated in step/operation 1101 in a highly sparse data object (e.g., ahighly sparse vector). In some embodiments, a highly sparse vector is avector whose sparsity exceeds a sparsity threshold. In some embodiments,to a corresponding encoding location for each appended training dataentry generated in step/operation 1101 in a highly sparse data object,the model generation unit 115 utilizes a hashing function configured todeterministically associate each appended training data entry generatedin step/operation 1101 to a particular encoding location in the highlysparse data object. For example, in some embodiments, the modelgeneration unit 115 may first convert each appended training data objectto a numeric value, e.g., based on the American Standard Code forInformation Interchange (ASCII) values for the characters of theappended training data object. Thereafter, the model generation unit 115may apply a hashing function to the numeric value to determine theencoding location.

At step/operation 1103, the model generation unit 115 stores eachappended training data object in the highly sparse data object based onthe encoding location for the appended training data object determinedin step/operation 1102. In some embodiments, the model generation unit115 identifies a particular location in the highly sparse data objectthat corresponds to the encoding location for a particular appendedtraining data object and stores the particular appended training dataobject in the particular location. In some embodiments, the modelgeneration unit 115 applies an offset to the encoding location for aparticular appended training data object to determine an offset-appliedlocation for the particular appended training data object, identifies aparticular location in in the highly sparse data object that correspondsto the offset-applied location for the particular appended training dataobject, and stores the particular appended training data object in theparticular location. In some embodiments, the model generation unit 115stores the highly sparse data object in the storage subsystem 108 (e.g.,as part of the structured input data 121 in the storage subsystem 108).

Returning to FIG. 5, at step/operation 505, the model generation unit115 updates the FTRL ML model based on each appended training dataobject generated in step/operation 504. In some embodiments, to updatethe FTRL ML model based on each appended training data object generatedin step/operation 504, the model generation unit 115 first retrieves theappended training data objects (e.g., form the highly sparse data objectgenerated in step/operation 1103, which may be stored as part of thestructured input data 121 in the storage subsystem 108). The modelgeneration unit 115 then performs a prediction on the appended trainingdata object based on the feature string included in the appendedtraining data object and using the FTRL ML model initialized instep/operation 502. Afterward, the model generation unit 115 determinesa measure of error between the prediction and the training labelincluded in the appended training object. The model generation unit 115then updates the FTRL ML model based on the measure of error (e.g.,using a gradient descent algorithm, such as an online gradient descentalgorithm). In some embodiments, to perform a prediction on the appendeddata object, the model generation unit 115 retrieves the modeldefinition data 123 for the FTRL ML model from the storage subsystem108. In some embodiments, after generating an updated FTRL ML model, themodel generation unit 115 stores the parameters and/or hyper-parametersassociated with the FTRL ML model as part of the model definition data123 for the FTRL ML model from the storage subsystem 108.

In some embodiments, to update the FTRL ML model parameters w for theFTRL ML model based on a measure of error g at a time step t, the modelgeneration unit 115 performs the operations corresponding to theequation w_(t+1)=w_(t)−ng_(t), where n is a non-increasing learningschedule rate value which may for example be set to 1/√t. In someembodiments, to update the FTRL ML model parameters w for the FTRL MLmodel based on a measure of error g at a time step t, the modelgeneration unit 115 performs the operations corresponding to theequation w_(t+1)=argmin_(w) (g_(1:t), w+0.5*Σ_(s=1) ^(t)σ_(s)∥w−w_(s)∥₂²+φ₁∥w∥₁), where σ may be a predictive parameter defined based on anon-increasing learning schedule rate value n such that σ_(1:t)=1/nt. Insome embodiments, to update the FTRL ML model parameters w for the FTRLML model based on a measure of error g at a time step t, the modelgeneration unit 115 performs the operations corresponding to theargmin_(w) (g_(1:t)−Σ_(s=1) ^(t)(σ_(s)w_(s))*w+1/nt∥w∥₂ ²+φ₁∥w∥₁), wheren is a non-increasing learning schedule rate value which may for examplebe set to 1/√t.

At step/operation 506, the model generation unit 115 determines whetherfurther FTRL ML model updates are needed. In some embodiments, todetermine whether further FTRL ML model updates are needed, the modelgeneration unit 115 first obtains the updated FTRL ML model generated instep/operation 505 (e.g., retrieves the parameters and/orhyper-parameters associated with the FTRL ML model from the modeldefinition data 123 for the FTRL ML model stored in the storagesubsystem 108). The model generation unit 115 then applies the updatedFTRL to one or more validation data objects and determines a predictionfor each validation data object. The model generation unit 115 thendetermines a validation score for the updated FTRL ML model based on thepredictions determined by the FTRL ML model for the one or morevalidation data objects and ground-truth validation labels for the oneor more validation data objects. The model generation unit 115 thendetermines whether further FTRL ML model updates are needed based on thevalidation score for the FTRL ML model (e.g., based on whether thevalidation score exceeds a predefined threshold validation score).

In some embodiments, step/operation 506 may be performed in accordancewith the various steps/operations depicted in FIG. 12, which is aflowchart diagram of an example process for validating an FTRL ML model.The example process depicted in FIG. 12 begins at step/operation 1201when the model generation unit 115 obtains the trained FTRL ML modelafter a particular training epoch (e.g., after a particular execution ofthe FTRL ML model updating discussed with respect to the step/operation505). In some embodiments, a training epoch may be determined as the setof updates to an FTRL ML model performed using a particular batch oftraining data. In some embodiments, a training epoch may be determinedas the set of updates to an FTRL performing using n training dataobjects, where n may be a preconfigured parameter and/or hyper-parameterof the FTRL ML model, a preconfigured parameter and/or hyper-parameterof the particular training algorithm utilized by the model generationunit 115 to train the FTRL ML model, a parameter and/or hyper-parameterof the FTRL ML model determined by using a ML algorithm, a parameterand/or hyper-parameter of the particular training algorithm utilized bythe model generation unit 115 to train the FTRL ML model which isdetermined by using a ML algorithm, and/or the like.

At step/operation 1202, the model generation unit 115 performs avalidation of the FTRL ML model using validation data to generate avalidation score for the particular training epoch. In some embodiments,to perform validation of the FTRL ML model using the validation data togenerate a validation score for the particular training epoch, the modelgeneration unit 115 first retrieves validation data entries (e.g., fromthe structured input data 121 on the storage subsystem 108 and/or fromthe raw training data 124 stored on the storage subsystem 108). In someembodiments, for each training data batch having n training dataentries, the model generation unit 115 designates i/n portion of thetraining data batch as training data utilized to train the FTRL ML modelduring a training epoch and j/m portion of the training data batch asvalidation data utilized to validate the FTRL ML model during avalidation epoch, where at least one of n, i, and j may be apreconfigured parameter and/or hyper-parameter of the FTRL ML model, apreconfigured parameter and/or hyper-parameter of the particulartraining algorithm utilized by the model generation unit 115 to trainthe FTRL ML model, a parameter and/or hyper-parameter of the FTRL MLmodel determined by using a ML algorithm, a parameter and/orhyper-parameter of the particular training algorithm utilized by themodel generation unit 115 to train the FTRL ML model which is determinedby using a ML algorithm, and/or the like.

In some embodiments, subsequent to retrieving validation data entries,the model generation unit 115 performs predictions corresponding to thevalidation data entries using the FTRL ML model. The model generationunit 115 may apply one or more parameters of the FTRL ML model to thevalidation data entries to determine a corresponding prediction for eachvalidation data entry. Thereafter, the model generation unit 115determines a measure of error between each prediction associated with acorresponding validation data object and a ground-truth label associatedwith the corresponding validation data object. In some embodiments, themodel generation unit 115 determines, for each validation data objectcharacterized by a ground-truth prediction label and associated with aninferred prediction generated based on the FTRL ML model, a measure ofnumeric deviation between the ground-truth prediction for the validationdata object and the associated inferred prediction for the validationdata object.

In some embodiments, subsequent to determining each measure of errorbetween a prediction associated with a corresponding validation dataobject and a ground-truth label associated with the correspondingvalidation data object, model generation unit 115 then determines avalidation score for the FTRL ML model based on an aggregation of eachnoted measure of error. In some embodiments, the model generation unit115 determines the validation score based on a measure of statisticaldistribution (e.g., a mean, median, mode, and/or the like.) of eachmeasure of error between a prediction associated with a correspondingvalidation data object and a ground-truth label associated with thecorresponding validation data object. For example, the model generationunit 115 may determine the validation score for the FTRL ML model basedon a measure of statistical distribution (e.g., a mean, median, mode,and/or the like.) of each measure of numeric deviation between aground-truth prediction for a validation data object and the associatedinferred prediction for a validation data object.

At step/operation 1203, the model generation unit 115 determines whetherthe validation score satisfies (e.g., exceeds and/or is equal to orgreater than) a threshold validation score. If the model generation unit115 determines that the validation score satisfies the thresholdvalidation score, the model generation unit 115 determines atstep/operation 1204 that no further updates to the FTRL ML model areneeded. However, if the model generation unit 115 determines that thevalidation score fails to satisfy the threshold validation score, themodel generation unit 115 determines at step/operation 1205 that furtherupdates to the FTRL ML model are needed. Thus, in some embodiments, themodel generation unit 115 continues the training for as many trainingepochs as needed until the FTRL ML model converges with validation dataentries and/or until the validation score determined for the FTRL MLmodel satisfies a threshold validation score.

In some embodiments, the threshold validation score is determined basedon at least one of a preconfigured parameter and/or hyper-parameter ofthe FTRL ML model, a preconfigured parameter and/or hyper-parameter ofthe particular training algorithm utilized by the model generation unit115 to train the FTRL ML model, a parameter and/or hyper-parameter ofthe FTRL ML model determined by using a ML algorithm, a parameter and/orhyper-parameter of the particular training algorithm utilized by themodel generation unit 115 to train the FTRL ML model which is determinedby using a ML algorithm, and/or the like. In some embodiments, thethreshold validation score is determined based on a measure ofstatistical distribution (e.g., a mean, median, mode, and/or the like.)of each measure of error between a prediction associated with acorresponding validation data object and a ground-truth label associatedwith the corresponding validation data object. In some embodiments, thethreshold validation score is determined based on a measure ofstatistical distribution (e.g., a mean, median, mode, and/or the like.)of each measure of numeric deviation between a ground-truth predictionfor a validation data object and the associated inferred prediction fora validation data object. In some embodiments, the threshold validationscore is the same value for multiple training epochs associated with anFTRL ML model. In some embodiments, the threshold validation score maybe different values for different training epochs associated with anFTRL ML model.

The example process depicted in FIG. 12 thus may lead to training of anFTRL ML model for as many training epochs as needed until the FTRL MLmodel converges with validation data entries and/or until the validationscore determined for the FTRL ML model satisfies a threshold validationscore. As noted above, however, this is only one example of variouspossible techniques for determining whether further updates to an FTRLML model are needed. Indeed, a person of ordinary skill in the art willrecognize that there are various other techniques for determiningwhether further updates to an FTRL ML model are needed, for example thetechnique of repeating the training of an FTRL ML model based on atraining repetition parameter p, which may be a statistically-determinedparameter or a dynamically-determined parameter. In some embodiments,the model generation unit 115 continues the training of the FTRL MLmodel for p training epochs. In some embodiments, the trainingrepetition parameter p may be determined based on at least one of apreconfigured parameter and/or hyper-parameter of the FTRL ML model, apreconfigured parameter and/or hyper-parameter of the particulartraining algorithm utilized by the model generation unit 115 to trainthe FTRL ML model, a parameter and/or hyper-parameter of the FTRL MLmodel determined by using a ML algorithm, a parameter and/orhyper-parameter of the particular training algorithm utilized by themodel generation unit 115 to train the FTRL ML model which is determinedby using a ML algorithm, and/or the like.

Returning to FIG. 5, if the model generation unit 115 determines atstep/operation 506 that further FTRL ML models are needed, the modelgeneration unit 115 repeats steps/operations 504 until after thesuccessful execution of a training epoch after which the modelgeneration unit 115 determines at step/operation 506 that no furtherFTRL ML models are needed. However, if the model generation unit 115determines at step/operation 506 that further FTRL ML models are notneeded, the model generation unit 115 exports the trained FTRL ML modelat step/operation 507. In some embodiments, after the model generationunit 115 determines at step/operation 506 that further FTRL ML modelsare not needed, the model generation unit 115 determines that the FTRLML model is a trained FTRL ML model and transmits the trained FTRL MLmodel to an end-user. In some embodiments, after the model generationunit 115 determines at step/operation 506 that further FTRL ML modelsare not needed, the model generation unit 115 determines that the FTRLML model is a trained FTRL ML model and stores the trained FTRL ML modelis a local and/or a remote database, e.g., as part of the modeldefinition data 123 for the FTRL ML model in the storage subsystem 108.

Once trained by the model generation unit 115, the online learning unit111 may utilize a trained online ML model (e.g., a trained FTRL MLmodel) to perform predictions in a hierarchical prediction domain. Toperform predictions in a hierarchical prediction domain using thetrained online ML model, the online learning unit 111 can apply thetrained online ML model to particular prediction inputs to generateprediction scores for each prediction node in the hierarchicalprediction domain. Thereafter, the online learning unit 111 can generatea final prediction output based on each prediction score for aprediction node in the hierarchical prediction domain. To do so, theonline learning unit 111 may perform a brute-force comparison of allprediction scores associated with a hierarchical prediction domain.This, however, may prove computationally expensive for largehierarchical prediction domains, such as the HPO hierarchical predictiondomain.

Accordingly, various embodiments of the present invention performpredictive inferences by selecting prediction nodes having sufficientlyhigh predictive scores starting from dependent prediction nodes. Indoing so, the mentioned embodiments of the present invention increasethe likelihood that prediction nodes having more detailed semanticimplications (e.g., more “meaningful” prediction nodes) will be selectedover prediction nodes having less detailed semantic associations, thusincreasing the reliability of the predictive analysis performed usingonline learning. For example, a prediction node associated with athoracolumbar scoliosis HPO label will have a higher chance of selectionthat a prediction node associated with a scoliosis HPO label, as theformer has a more meaningful semantic association than the former. Thiswill lead to generation of structurally hierarchical predictions whichhave greater predictive utility. Moreover, selection of predictionlabels in a hierarchical manner decreases the range of predictive scoresthat need to be analyzed during a predictive inference. This is because,according to various embodiments of the present invention, predictiveinference will halt if a requisite number of prediction nodes areselected inmost-dependent nodes. Such techniques have the addedadvantage of increasing efficiency of online learning in hierarchicalprediction domains by decreasing the range of prediction nodes whichneed to be traversed before a final prediction output is generated aspart of a particular predictive inference. Thus, by selecting predictionnodes having sufficiently high predictive scores starting from dependentprediction nodes, various embodiments of the present invention addressefficiency and reliability challenges related to utilizing onlinelearning algorithms to generate predictions related to hierarchicalprediction domains.

FIG. 13 provides a flowchart diagram of an example process 1300 forgenerating prediction labels using a trained FTRL ML model and in ahierarchical prediction domain. Via the various steps/operations ofprocess 1300, the online learning unit 111 of the classificationcomputing entity 106 can perform predictive inferences by selectingprediction nodes having sufficiently high predictive scores startingfrom dependent prediction nodes in a manner that improves efficiency andreliability of applying the FTRL ML model to generate predictionsrelated to hierarchical prediction domains.

The process 1300 begins at step/operation 1301 when the online learningunit 111 obtains a prediction input data object. The prediction inputdata object may be a data object that includes one or more predictioninput features, such as one or more structured input features (e.g.,medical codes for a particular patient). In some embodiments, theprediction input data object is associated with a predictive entity(e.g., a patient), which may be a real-world entity with respect towhich a prediction inference is being made. In some embodiments, atleast a portion of the prediction input data object is obtained from anexternal computing entity 102, such as an external computing entity 102associated with a healthcare delivery organization, an externalcomputing entity 102 associated with a health insurance providerorganization, an external computing entity 102 associated with anauditing organization, an external computing entity 102 associated witha regulatory organization, and/or the like. In some embodiments, atleast a portion of the prediction input data object is retrieved from alocal and/or remote database, such as from the storage subsystem 108 ofthe classification system 101. In some embodiments, the prediction inputdata object is a prediction input vector, e.g., an atomic m*1 vectorstored as a one-dimensional array of size m and a vector in a predictioninput in an m*n matrix including n vectors stored as a two-dimensionalarray of size m*n.

At step/operation 1302, the online learning unit 111 obtains a trainedFTRL ML model, e.g., a trained FTRL ML model trained using the exampleprocess 500 depicted in FIG. 5. In some embodiments, to obtain thetrained FTRL ML model, the online learning unit 111 retrieves parametersand/or hyper-parameters associated with the FTRL ML model from the modeldefinition data 123 in the storage subsystem 108. In some embodiments,the FTRL ML model is an FTRL ML model trained without a bias term and/orwith initial weight values configured to generate negative predictionscores. Although the example process 1300 depicted in FIG. 13 discussesan FTRL ML model, a person of ordinary skill in the art will recognizethat the process 1300 may be performed using any ML model, such as anyonline ML model.

At step/operation 1303, the online learning unit 111 utilizes thetrained FTRL ML model to generate prediction score for each predictionnode in the hierarchical prediction domain. In some embodiments, thetrained FTRL ML model may be configured such that, for each hierarchicalstructure within the hierarchical prediction domain which includes agroup of prediction nodes having one or more head prediction nodes, themore hierarchically dependent prediction nodes have higher predictionscores than the less hierarchically dependent prediction nodes. Forexample, given a hierarchical prediction structure characterized by thehierarchical prediction relationships A»B,C; B»D, E; C»F; and F»I (whereX»Y,Z denotes that prediction nodes Y and Z are hierarchically dependenton the prediction node X); the prediction node I may have a higherprediction score than the prediction nodes F and C and A, the predictionnode F may have a higher prediction score than the prediction nodes Cand A, the prediction scores D and E may have a higher prediction scorethan the prediction nodes B and A, and the predictions B and C may havea higher prediction score than the prediction node A. In otherembodiments, the trained FTRL ML model may be configured to generateprediction scores irrespective of the hierarchical position of aprediction node in the hierarchical prediction domain.

FIG. 14 provides an operational example of a predictive score dataobject 1400 for a hierarchical structure within a hierarchicalprediction domain. The hierarchical structure corresponding to thepredictive score data object 1400 includes a group of sixteen predictionnodes with a head prediction node 1409, from which the fifteen non-headprediction nodes directly or indirectly depend. The predictive scoredata object 1400 includes a predictive score for each prediction nodeassociated with the hierarchical structure, including the followingprediction scores: a prediction score 0.5 for dependent prediction node1401; prediction scores 0.9, 0.3, and 0.1 for second-layer predictionnodes 1402, 1403, and 1404; prediction scores 0.2, 0.4, 0.5, and 0.9 forthird-layer prediction nodes 1405, 1406, 1407, and 1408; and predictionscore 0.4 for the head prediction node 1409.

Returning to FIG. 13, at step/operation 1304, the online learning unit111 selects up to K prediction labels on each hierarchical level,starting from the most-dependent hierarchical level. At step/operation1305, the online learning unit 111 determines whether K predictionlabels have been selected from all hierarchical levels. In someembodiments, starting at the most-dependent hierarchical level and goingto the higher hierarchical levels, the online learning unit 111 selectstop prediction nodes on each layer until it reaches K prediction nodes.Therefore, a prediction node on a lower hierarchical level may beselected over a prediction node on an upper hierarchical level eventhough the former prediction node has a lower prediction score than thelatter prediction node. In some embodiments, K is an output diversityparameter for the FTRL ML model. In some embodiments, the outputdiversity parameter for the FTRL ML model may be determined based on atleast one of a preconfigured parameter and/or hyper-parameter of theFTRL ML model, a preconfigured parameter and/or hyper-parameter of theparticular training algorithm utilized by the model generation unit 115to train the FTRL ML model, a parameter and/or hyper-parameter of theFTRL ML model determined by using a ML algorithm, a parameter and/orhyper-parameter of the particular training algorithm utilized by themodel generation unit 115 to train the FTRL ML model which is determinedby using a ML algorithm, and/or the like.

In some embodiments, step/operation 1304 may be described using theoperational example of FIG. 14. For example, if K=3, the online learningunit 111 may select the following prediction nodes from the hierarchicalstructure associated with the prediction score data object 1400 of FIG.14: prediction node 1401 (i.e., the prediction node having the highestprediction score in the most-dependent hierarchical level, whoseselection increments a node selection count value I to generate I=1);prediction node 1403 (i.e., the prediction node having the highestprediction score in the second hierarchical level, whose selectionincrements I to generate I=2); and prediction node 1404 (i.e., theprediction node having the second-highest prediction score in the secondhierarchical level, whose selection increments I to generate I=3, whichequals K thus terminating the graph traversal).

By selecting prediction nodes based at least in part on the hierarchicallevel of those prediction nodes, various embodiments of the presentinvention perform predictive inferences by selecting prediction nodeshaving sufficiently high predictive scores starting from dependentprediction nodes. In doing so, the mentioned embodiments of the presentinvention increase the likelihood that prediction nodes having moredetailed semantic implications (e.g., more “meaningful” predictionnodes) will be selected over prediction nodes having less detailedsemantic associations, thus increasing the reliability of the predictiveanalysis performed using online learning. For example, a prediction nodeassociated with a thoracolumbar scoliosis HPO label will have a higherchance of selection that a prediction node associated with a scoliosisHPO label, as the former has a more meaningful semantic association thanthe former. This will lead to generation of structurally hierarchicalpredictions which have greater predictive utility. Moreover, selectionof prediction labels in a hierarchical manner decreases the range ofpredictive scores that need to be analyzed during a predictiveinference. This is because, according to various embodiments of thepresent invention, predictive inference will halt if a requisite numberof prediction nodes are selected among the most-dependent nodes. Suchtechniques have the added advantage of increasing efficiency of onlinelearning in hierarchical prediction domains by decreasing the range ofprediction nodes which need to be traversed before a final predictionoutput is generated as part of a particular predictive inference. Thus,by selecting prediction nodes having sufficiently high predictive scoresstarting from dependent prediction nodes, various embodiments of thepresent invention address efficiency and reliability challenges relatedto utilizing online learning algorithms to generate predictions relatedto hierarchical prediction domains.

Moreover, by selecting prediction nodes based at least in part on thehierarchical level of those prediction nodes, various embodiments of thepresent invention generate structurally hierarchical predictions. Astructurally hierarchical prediction may be a prediction determinedbased at least in part on a position of a corresponding prediction nodein a structural hierarchy characterizing the hierarchical predictiondomain that includes the corresponding prediction node. For example, insome embodiments, an online ML model may be configured to generate theone or more online learning predictions based on prediction values forprediction nodes deemed to be dependent prediction nodes in astructurally hierarchy characterizing the prediction domain associatedwith the online ML model. In at least some of those embodiments, theonline ML model will not generate predictions that correspond tonon-most-dependent nodes in the structural hierarchically characterizingthe prediction domain associated with the online ML model. By generatingstructurally hierarchical predictions, the online learning unit 111 cangenerate prediction nodes based on prediction nodes deemed to be moresemantically significant. Thus, by selecting prediction nodes based atleast in part on the hierarchical level of those prediction nodes,various embodiments of the present invention improve reliability ofonline ML in hierarchical prediction domains, such as in hierarchicalprediction domains having a large number of hierarchical structuresand/or having a large number of prediction nodes.

Returning to FIG. 13, if the online learning unit 111 determines atstep/operation 1305 that K prediction labels have been selected from allhierarchical layers in the hierarchical structure, the online learningunit 111 generates the prediction labels based on the selected Kprediction labels. In some embodiments, the online learning unit 111provides the prediction labels and/or a prediction output determinedbased on the prediction labels to an external computing entity 102, suchas an external computing entity 102 associated with a healthcaredelivery organization, an external computing entity 102 associated witha health insurance provider organization, an external computing entity102 associated with an auditing organization, an external computingentity 102 associated with a regulatory organization, and/or the like.In some embodiments, the online learning unit 111 stores the predictionlabels and/or a prediction output determined based on the predictionlabels on a local and/or remote database, such as from the storagesubsystem 108 of the classification system 101.

However, if the online learning unit 111 determines at step/operation1305 that K prediction labels have not been selected from allhierarchical layers in the hierarchical structure, the online learningunit proceeds to a higher layer and performs the step/operation 1304with respect to that layer. Thus, if for example K=5, the onlinelearning unit 111 may select the following prediction nodes from thehierarchical structure associated with the prediction score data object1400 of FIG. 14: prediction node 1401 (i.e., the prediction node havingthe highest prediction score in the most-dependent hierarchical level,whose selection increments a node selection count value I to generateI=1); prediction node 1403 (i.e., the prediction node having the highestprediction score in the second hierarchical level, whose selectionincrements I to generate I=2); prediction node 1404 (i.e., theprediction node having the second-highest prediction score in the secondhierarchical level, whose selection increments I to generate I=3);prediction node 1408 (i.e., the prediction node having the highestprediction score in the third hierarchical level, whose selectionincrements I=4); and prediction node 1407 (i.e., the prediction nodehaving the second-highest prediction score in the third hierarchicallevel, whose selection increments I=5, which equals K and thusterminates the graph traversal).

C. Co-Occurrence Analysis in Hierarchical Prediction Domains

While online ML provides important insights about relationships amongprediction inputs with hierarchical structures in a hierarchicalprediction domains and has the flexibility of sequential updatabilityover time, other important insights can be inferred from analyzingstatistical relations of particular features and particular predictionlabels among training data. However, given large amounts of trainingdata, such statistical analyses may suffer from reliability drawbacks ifthey do not properly accommodate for factors that complicate conceptualpredictive inferences from numeric patterns. For example, triviallyfrequent correlations can complicate accurate and reliable conceptualinferences from statistical correlations. As another example, markedlyinfrequent occurrences can also complicate statistical analysis ofpredictive data in order to infer conceptual notions that can facilitateeffective classification. As a further example, conceptually obviouscorrelations may distort cross-data analyses of correlations betweenfeatures and prediction labels without contributing sufficientconceptual value to the predictive inference process.

Because of such complexities associated with translation of numericpatterns to conceptual predictive frameworks, many existing statisticalML problems face substantial challenges when it comes to efficiently andreliably performing predictive inferences based on co-occurrence data.To address reliability concerns stemming from complexities associatedwith translation of numeric patterns to conceptual predictiveframeworks, many conventional statistical ML problems resort toexpensive training operations that undermine efficiency of ML solutionswithout sufficiently contributing to the reliability and accuracy of thepredictions performed by those ML solutions. Thus, there is a continuingtechnical need for efficient and reliable solutions for statistical MLin various classification domains, such as in hierarchical predictiondomains.

Various embodiments of the present invention address the efficiency andreliability challenges related to complexities associated withtranslation of numeric patterns to conceptual predictive frameworks. Forexample, various embodiments of the present invention provide innovativesolutions for both normalizing feature-label co-occurrence data andsignificance-based filtering of such co-occurrence data. Through thenoted techniques, various embodiments of the present invention providecomputationally efficient solutions that address complexities associatedwith translation of numeric patterns to conceptual predictiveframeworks, such as complexities associated with trivially frequentco-occurrences, complexities associated with mistakenly under-recordedco-occurrences, and complexities associated with conceptually obviousco-occurrences. Accordingly, by both normalizing feature-labelco-occurrence data and significance-based filtering of suchco-occurrence data, various embodiments of the present invention addresstechnical challenges related to efficiency and reliability ofstatistical ML solutions and improve efficiency and reliability ofvarious existing conventional statistical ML solutions. The resultingimprovements address efficiency and reliability of all statistical MLsolutions, including statistical ML solutions utilized in hierarchicalprediction domains. Thus, while aspects of the co-occurrence analysis MLmodels are described herein with respect to statistical ML solutionsutilized in hierarchical prediction domains, one of ordinary skill inthe art will recognize that the co-occurrence analysis ML models can beutilized to improve efficiency and reliability of all statistical MLsolutions, including statistical ML solutions utilized innon-hierarchical prediction domains.

In addition to improving efficiency and reliability of all statisticalML solutions, some aspects of the co-occurrence analysis ML modelsdescribed herein include important contributions to efficiency andreliability of ML in hierarchical prediction domains. In hierarchicalprediction domains, the presence of hierarchical relationships betweenprediction nodes in the output space complicates the task of inferring aprediction output based on prediction scores for various predictionnodes. On the one hand, the hierarchical relationships betweenprediction nodes in the output space provide important domaininformation that can facilitate efficient and reliable predictiveinferences. On the other hand, important predictive conclusions may beinferred from ignoring the hierarchical relationships, especially ininstances where the available hierarchical models do not capture all ofthe relevant information about conceptual relationships betweenprediction nodes and/or include potentially erroneous information aboutconceptual relationships between prediction nodes. Thus, there is acontinuing technical challenge associated with performing predictiveanalyses in a manner that takes into account both hierarchicalcomposition of the output space and cross-hierarchical composition ofthe output space.

Various embodiments of the present invention address the mentionedtechnical challenges associated with considering both hierarchicalcomposition of the output space and cross-hierarchical composition ofthe output space when performing classification in hierarchicalprediction domains. For example, various embodiments of the presentinvention relate to co-occurrence analysis ML models that generatestructurally non-hierarchical predictions. A structurallynon-hierarchical prediction may be a prediction determined withoutregard to a position of the corresponding prediction node in astructural hierarchy characterizing the hierarchical prediction domainthat includes the corresponding prediction node. For example, in someembodiments, a co-occurrence analysis ML model may be configured togenerate the one or more co-occurrence analysis predictions based onprediction values for prediction nodes regardless of whether theprediction nodes are deemed to be dependent prediction nodes in astructurally hierarchy characterizing the prediction domain associatedwith the co-occurrence analysis ML model. In at least some of thoseembodiments, the co-occurrence analysis ML model may generatepredictions that correspond to both most-dependent nodes andnon-most-dependent nodes in the structural hierarchically characterizingthe prediction domain associated with the online ML model.

By generating structurally non-hierarchical predictions, variousembodiments of the present invention provide predictions that areagnostic to the hierarchical composition of the prediction output space.When used in combination and/or in fusion with structurally hierarchicalpredictions (e.g., online learning predictions generated by an online MLmodel), such predictions can provide important cross-hierarchicalconceptual inferences that can in turn facilitate efficient andeffective classification in conceptually hierarchical domains. Thus, bygenerating structurally non-hierarchical predictions that can in turn beused in combination and/or in fusion with structurally hierarchicalpredictions, various embodiments of the present invention addresstechnical challenges related to accounting for both hierarchicalcomposition of the output space and cross-hierarchical composition ofthe output space when performing classification in hierarchicalprediction domains. In doing so, various embodiments of the presentinvention make important technical contributions to efficiency andreliability of classification in hierarchical prediction domains, suchas in classification in an HPO prediction domain.

FIG. 15 is a flowchart diagram of an example process 1500 for training aco-occurrence analysis ML model. Via the various steps/operations ofprocess 1500, the model generation unit 115 of the classificationcomputing entity 106 can train a co-occurrence analysis ML modelconfigured to generate/predict accurate and reliable structurallynon-hierarchical predictions based on prediction inputs associated witha hierarchical prediction space.

The process 1500 begins at step/operation 1501 when the model generationunit 115 prepares input training data. In some embodiments, to preparethe input training data, the model generation unit 115 retrieves rawtraining data 124 from the storage subsystem 108. The raw training data124 may include, for each training entity (e.g., patient), one or moretraining features (e.g., medical codes) and one or more trainingprediction labels (e.g., one or more HPO labels), where at least one ofthe training prediction labels may be associated with a hierarchicalpredictive relationship.

At step/operation 1502, the model generation unit 115 constructs aco-occurrence matrix based on the input training data obtained instep/operation 1501. In some embodiments, each value in theco-occurrence matrix is determined based on a count of co-occurrence ofa respective training feature and a respective prediction label in theinput training data. For example, given the input data objects {B1, B2,B3→Y1}, {B1, B4, B5→Y1}, {B1, B3, B4→Y2}, {B1, B2, B5→Y2}, and {B1, B2,B4→Y2} (where {α, β, ç→{acute over (ε)}} denotes that training featuresα, β, ç are associated with the training prediction label {acute over(ε)}), the model generation unit 115 may generate the followingco-occurrence matrix values (where M(α, {acute over (ε)}) denotes aco-occurrence matrix value associated with the training feature a andthe training prediction label {acute over (ε)}): M(B1, Y1)=2; M(B2,Y1)=1; M(B3, Y1)=1; M(B4, Y1)=1; M(B5, Y1)=1; M(B1, Y2)=3; M(B2, Y1)=2;M(B3, Y2)=1; M(B4, Y2)=2; and M(B5, Y2)=1.

FIG. 16 provides an operational example of a co-occurrence matrix 1600.In the example co-occurrence matrix 1600 of FIG. 16, each co-occurrencematrix value represents a magnitude (e.g., count) of co-occurrence of aparticular training feature (i.e., an ICD value, represented by thecolumns 1601 of the co-occurrence matrix 1600) and a particular trainingprediction label (i.e., an HPO label, represented by the rows 1602 ofthe co-occurrence matrix). For example, based on the co-occurrencematrix 1600, the model generation unit 115 can infer that the magnitudeof co-occurrence of training feature ICD-2 and HPO label HPO-20 is 82.As another example, based on the co-occurrence matrix 1600, the modelgeneration unit 115 can infer that the magnitude of co-occurrence oftraining feature ICD-7 and HPO label HPO-20 is 92. As yet anotherexample, based on the co-occurrence matrix 1600, the model generationunit 115 can infer that the magnitude of co-occurrence of trainingfeature ICD-12 and HPO label HPO-20 is 59. As a further example, basedon the co-occurrence matrix 1600, the model generation unit 115 caninfer that the magnitude of co-occurrence of training feature ICD-15 andHPO label HPO-20 is 28.

Returning to FIG. 15, at step/operation 1503, the model generation unit115 performs one or more normalizations of the co-occurrence matrixgenerated in step/operation 1502 in order to generate a normalizedco-occurrence matrix. In some embodiments, the model generation unit 115performs at least one of row-wide normalizations and column-widenormalizations to perform the one or more normalizations of theco-occurrence matrix generated in step/operation 1502 in order togenerate the mentioned normalized co-occurrence matrix. In someembodiments, the model generation unit 115 may perform normalizationsacross each group of consecutive n rows and/or each group of consecutivem columns, where at least one of n and m may be more than one. In someembodiments, at least one of n and m may be a preconfigured parameterand/or hyper-parameter of the co-occurrence analysis ML model, apreconfigured parameter and/or hyper-parameter of the particulartraining algorithm utilized by the model generation unit 115 to trainthe co-occurrence analysis ML model, a parameter and/or hyper-parameterof the co-occurrence analysis ML model determined by using a MLalgorithm, a parameter and/or hyper-parameter of the particular trainingalgorithm utilized by the model generation unit 115 to train theco-occurrence analysis ML model which is determined by using a MLalgorithm, and/or the like.

In some embodiments, step/operation 1503 may be performed in accordancewith the various steps/operations of the FIG. 17, which is a flowchartdiagram of an example process for generating a normalized co-occurrencematrix. The example process depicted in FIG. 17 begins at step/operation1701 when the model generation unit 115 obtains a co-occurrence matrix,e.g., the co-occurrence matrix 1600 of FIG. 16. At step/operation 1702,the model generation unit 1702 performs one or more row-widenormalizations of the co-occurrence matrix to generate a row-normalizedco-occurrence matrix. In some embodiments, to perform a row-widenormalization of a particular row of the co-occurrence matrix, the modelgeneration unit 115 applies a normalization parameter to eachco-occurrence matrix value in the particular row, where thenormalization parameter is determined based on a measure of statisticaldistribution (e.g., median, mean, mode, maximum value, minimum value,and/or the like.) of at least some co-occurrence matrix values in theparticular row. In some embodiments, to perform a row-wide normalizationof a particular row of the co-occurrence matrix, the model generationunit 115 divides each co-occurrence matrix value in the particular rowby a measure of statistical distribution (e.g., median, mean, mode,maximum value, minimum value, and/or the like.) of the co-occurrencematrix values in the particular row. For example, to perform a row-widenormalization of a particular row of the co-occurrence matrix, the modelgeneration unit 115 may divide each co-occurrence matrix value in theparticular row by a maximum co-occurrence matrix value in the particularrow. In some embodiments, to perform a row-wide normalization of aparticular row of the co-occurrence matrix, the model generation unit115 performs a softmax normalization of the co-occurrence matrix valuesin the particular row of the particular co-occurrence matrix.

At step/operation 1703, the training performs one or more column-widenormalizations of the row-normalized co-occurrence matrix generated instep/operation 1702 to generate a normalized co-occurrence matrix. Insome embodiments, to perform a column-wide normalization of a particularcolumn of the row-normalized co-occurrence matrix, the model generationunit 115 applies a normalization parameter to each row-normalizedco-occurrence matrix value in the particular row, where thenormalization parameter is determined based on a measure of statisticaldistribution (e.g., median, mean, mode, maximum value, minimum value,and/or the like.) of at least some row-normalized co-occurrence matrixvalues in the particular column. In some embodiments, to perform acolumn-wide normalization of a particular column of the co-occurrencematrix, the model generation unit 115 divides each row-normalizedco-occurrence matrix value in the particular column by a measure ofstatistical distribution (e.g., median, mean, mode, maximum value,minimum value, and/or the like.) of the row-normalized co-occurrencematrix values in the particular column. For example, to perform acolumn-wide normalization of a particular column of the row-normalizedco-occurrence matrix, the model generation unit 115 may divide eachco-occurrence matrix value in the particular column by a maximumrow-normalized co-occurrence matrix value in the particular column. Insome embodiments, to perform a column-wide normalization of a particularcolumn of the row-normalized co-occurrence matrix, the model generationunit 115 performs a softmax normalization of the row-normalizedco-occurrence matrix values in the particular column of therow-normalized co-occurrence matrix.

Returning to FIG. 15, at step/operation 1504, the model generation unit115 performs one or more significance-based filters of the normalizedco-occurrence matrix generated in step/operation 1503 in order togenerate a filtered co-occurrence matrix. In some embodiments, the toperform a significance-based filter of the normalized co-occurrencematrix, the model generation unit 115 may compute a non-parametricadjustment value for each normalized co-occurrence matrix value thatdescribes at least one aspect of the significance of the normalizedco-occurrence matrix value in its respective category (e.g., in itsrespective row and/or column of the normalized co-occurrence matrix)relative to the significance of other normalized co-occurrence matrixvalues in other respective categories. An example of asignificance-based filter is a filter performed based on a chi-squaredtest analysis. While the example process 1500 depicts thesignificance-based filters of the co-occurrence matrix as beingperformed after co-occurrence matrix normalizations, one of ordinaryskill in the art will recognize that some significance-based filters canbe performed before co-occurrence matrix normalizations while othersignificance-based filters can be performed after co-occurrence matrixnormalizations, and/or the like. In general, the model generation unit115 may perform each of various co-occurrence matrix normalizations andvarious co-occurrence matrix significance-based filters in any order togenerate a co-occurrence matrix as part of training a co-occurrenceanalysis ML model associated with the co-occurrence analysis unit 112.In some embodiments, the model generation unit 115 may utilize atraining algorithm (e.g., a gradient method algorithm that utilizes anerror function generated based on prediction scores determined withrespect to ground-truth prediction data) to generate weights for each ofvarious entries in a co-occurrence matrix and adjust the notedco-occurrence matrix entries according to their corresponding weightvalues.

Once trained by the model generation unit 115, the co-occurrenceanalysis ML model can be transmitted to an end user, e.g., an end-userassociated with an external computing entity, such as an externalcomputing entity 102 associated with a healthcare delivery organization,an external computing entity 102 associated with a health insuranceprovider organization, an external computing entity 102 associated withan auditing organization, an external computing entity 102 associatedwith a regulatory organization, and/or the like. Moreover, once trainedby the model generation unit 115, the co-occurrence analysis ML modelcan be stored in a local and/or remote database, e.g., as part of themodel definition data 123 for the co-occurrence analysis ML model in thestorage subsystem 108.

The trained co-occurrence analysis ML model can be used by theco-occurrence analysis unit 112 of the classification computing entity106 to generate predictions, such as predictions related to ahierarchical prediction domain. The normalization and filteringoperations performed on the co-occurrence data by the model generationunit 115 are examples of computationally inexpensive operationsconfigured to address complexities associated with translation ofnumeric patterns to conceptual predictive frameworks. Moreover, theco-occurrence model utilized by the co-occurrence analysis unit 112 maybe configured to generate structurally non-hierarchical predictions thatprovide important cross-hierarchical insights that facilitate effectiveand efficient classification in hierarchical prediction domains.

FIG. 18 is a flowchart diagram of an example process 1800 for generatingpredictions using a trained co-occurrence analysis ML model. Via thevarious steps/operations of the process 1800, the co-occurrence analysisunit 112 of the classification computing entity 106 can generatestructurally non-hierarchical predictions that facilitate effective andefficient classification in hierarchical prediction domains.

Process 1800 begins at step/operation 1801 when the co-occurrenceanalysis unit 112 obtains a trained co-occurrence analysis ML model. Insome embodiments, to obtain the trained co-occurrence analysis ML model,the co-occurrence analysis unit 112 retrieves parameters and/orhyper-parameters associated with the co-occurrence analysis ML modelfrom the model definition data 123 associated with the co-occurrenceanalysis ML model in the storage subsystem 108. In some embodiments, theco-occurrence analysis ML model is generated based on the co-occurrencesof training features and training prediction labels among particulartraining data, e.g., particular raw training data 124 stored in thestorage subsystem 108. In some embodiments, the co-occurrence analysisML model is generated using a training process that involves one or morematrix normalizations and/or one or more matrix significance-basedfilters, e.g., the example process 1500 of FIG. 15.

At step/operation 1802, the co-occurrence analysis unit 112 obtainsparticular one or more prediction input features. In some embodiments,the particular prediction input features are associated with aparticular predictive entity, which may be a real-world entity withrespect to which a prediction is being performed. Examples of particularprediction input features for a particular predictive entity may bemedical codes associated with a particular patient. In some embodiments,at least a portion of the particular prediction input features isobtained from an external computing entity 102, such as an externalcomputing entity 102 associated with a healthcare delivery organization,an external computing entity 102 associated with a health insuranceprovider organization, an external computing entity 102 associated withan auditing organization, an external computing entity 102 associatedwith a regulatory organization, and/or the like. In some embodiments, atleast a portion of the particular prediction input features is retrievedfrom a local and/or remote database, such as from the storage subsystem108 of the classification system 101. In some embodiments, at least aportion of the particular prediction input features is stored in aprediction input data object. In some embodiments, the prediction inputdata object is a prediction input vector, e.g., an atomic m*1 vectorstored as a one-dimensional array of size m and a vector in a predictioninput in an m*n matrix including n vectors stored as a two-dimensionalarray of size m*n.

At step/operation 1803, the co-occurrence analysis unit 112 determinestop M prediction labels for the prediction input features obtained instep/operation 1802. In some embodiments, to determine the top Mprediction labels for the prediction input features obtained instep/operation 1802, the co-occurrence analysis unit 112 uses thetrained co-occurrence matrix. For example, the co-occurrence analysisunit 112 may determine the top M prediction labels having the highestco-occurrence matrix values associated with a particular predictioninput feature in the trained co-occurrence matrix obtained instep/operation 1801 as the top M prediction labels for the particularprediction input feature. In some embodiment, if the prediction inputfeatures obtained in step/operation 1802 include more than one features,the co-occurrence analysis unit 112 may determine the top M predictionlabels having the highest co-occurrence matrix values associated withthe at least one of the two or more particular prediction input featuresin the trained co-occurrence matrix obtained in step/operation 1801 asthe top M prediction labels for the particular prediction input feature.

In some embodiments, M is an output diversity parameter for theco-occurrence analysis ML model. In some embodiments, the outputdiversity parameter for the co-occurrence analysis ML model may be apreconfigured parameter and/or hyper-parameter of the co-occurrenceanalysis ML model, a preconfigured parameter and/or hyper-parameter ofthe particular training algorithm utilized by the model generation unit115 to train the co-occurrence analysis ML model, a parameter and/orhyper-parameter of the co-occurrence analysis ML model determined byusing a ML algorithm, a parameter and/or hyper-parameter of theparticular training algorithm utilized by the model generation unit 115to train the co-occurrence analysis ML model which is determined byusing a ML algorithm, and/or the like.

FIG. 19 provides an operational example of a co-occurrence values set1900 for a training feature ICD-7. The co-occurrence values set 1900include twenty co-occurrence values that each denote a magnitude ofco-occurrence of the training feature ICD-7 with each of twenty trainingprediction labels, e.g., a normalized and/or significance-based filteredmagnitude of co-occurrence of the training feature ICD-7 with each oftwenty training prediction labels. The twenty co-occurrence values inthe co-occurrence values set 1900 include a co-occurrence value 1901(i.e., denoting a magnitude of co-occurrence of 96 between the trainingfeature ICD-7 and a corresponding prediction label), a co-occurrencevalue 1902 (i.e., denoting a magnitude of co-occurrence of 92 betweenthe training feature ICD-7 and a corresponding prediction label), aco-occurrence value 1903 (i.e., denoting a magnitude of co-occurrence of84 between the training feature ICD-7 and a corresponding predictionlabel), and a co-occurrence value 1904 (i.e., denoting a magnitude ofco-occurrence of 94 between the training feature ICD-7 and acorresponding prediction label).

In some embodiments, step/operation 1803 may be described with referenceto the exemplary co-occurrence values set 1900 of FIG. 19. If theco-occurrence analysis unit 112 obtains a singular prediction inputfeature corresponding to ICD-17 in step/operation 1802, theco-occurrence analysis unit 112 may determine the top M predictionlabels having the highest co-occurrence values as the prediction labelsfor the particular singular prediction input feature corresponding toICD-17. For example, given M=1, the co-occurrence analysis unit 112 maydetermine the prediction label corresponding to the co-occurrence value1901 as the selected prediction label for the particular singularprediction input feature, given that the co-occurrence value 1901 is thehighest co-occurrence value in the co-occurrence values set 1900. Asanother example, given M=2, the co-occurrence analysis unit 112 maydetermine the prediction label corresponding to the co-occurrence values1901 and 1902 as the selected prediction labels for the particularsingular prediction input feature, given that the co-occurrence values1901 and 1902 are the two highest co-occurrence values in theco-occurrence values set 1900. As yet another example, given M=3, theco-occurrence analysis unit 112 may determine the prediction labelcorresponding to the co-occurrence values 1901, 1902 and 1903 as theselected prediction labels for the particular singular prediction inputfeature, given that the co-occurrence values 1901, 1902 and 1903 are thethree highest co-occurrence values in the co-occurrence values set 1900.As a further example, given M=4, the co-occurrence analysis unit 112 maydetermine the prediction label corresponding to the co-occurrence values1901, 1902, 1903, and 1904 as the selected prediction labels for theparticular singular prediction input feature, given that theco-occurrence values 1901, 1902, 1903, and 1904 are the four highestco-occurrence values in the co-occurrence values set 1900.

Importantly, selecting the prediction labels based on the co-occurrencevalues in the trained co-occurrence matrix requires a computationallyinexpensive matrix traversal. Moreover, the generated prediction labelsare structurally non-hierarchical predictions, given that they may befrom any hierarchical level in a hierarchical prediction domains. Usingthese and other techniques, the co-occurrence analysis conceptsdiscussed herein provide efficient and reliable solutions forclassification using statistical ML solutions and/or classification inhierarchical prediction domains.

D. Fusion of Structurally Hierarchical Predictions and StructurallyNon-Hierarchical Predictions

As discussed above with reference to co-occurrence analysis ML models,in hierarchical prediction domains, the presence of hierarchicalrelationships between prediction nodes in the output space complicatesthe task of inferring a prediction output based on prediction scores forvarious prediction nodes. On the one hand, the hierarchicalrelationships between prediction nodes in the output space provideimportant domain information that can facilitate efficient and reliablepredictive inferences. On the other hand, important predictiveconclusions may be inferred from ignoring the hierarchicalrelationships, especially in instances where the available hierarchicalmodels do not capture all of the relevant information about conceptualrelationships between prediction nodes and/or include potentiallyerroneous information about conceptual relationships between predictionnodes. Thus, there is a continuing technical challenge associated withperforming predictive analyses in a manner that takes into account bothhierarchical composition of the output space and cross-hierarchicalcomposition of the output space.

Various embodiments of the present invention address the mentionedtechnical challenges associated with considering both hierarchicalcomposition of the output space and cross-hierarchical composition ofthe output space when performing classification in hierarchicalprediction domains. For example, various embodiments of the presentinvention relate to co-occurrence analysis ML models that generatestructurally non-hierarchical predictions. A structurallynon-hierarchical prediction may be a prediction determined withoutregard to a position of the corresponding prediction node in astructural hierarchy characterizing the hierarchical prediction domainthat includes the corresponding prediction node. For example, in someembodiments, a co-occurrence analysis ML model may be configured togenerate the one or more co-occurrence analysis predictions based onprediction values for prediction nodes regardless of whether theprediction nodes are deemed to be dependent prediction nodes in astructurally hierarchy characterizing the prediction domain associatedwith the co-occurrence analysis ML model. In at least some of thoseembodiments, the co-occurrence analysis ML model may generatepredictions that correspond to both most-dependent nodes andnon-most-dependent nodes in the structural hierarchically characterizingthe prediction domain associated with the online ML model.

By generating structurally non-hierarchical predictions, variousembodiments of the present invention provide predictions that areagnostic to the hierarchical composition of the prediction output space.Such structurally non-hierarchical predictions can in turn be used incombination and/or in fusion with structurally hierarchical predictions,such as structurally hierarchical predictions generated by an onlinelearning unit 111. When structurally non-hierarchical predictions areused in combination and/or in fusion with structurally hierarchicalpredictions are used to generate structure-based predictions, suchstructured-fused predictions can provide important cross-hierarchicalconceptual inferences that can in turn facilitate efficient andeffective classification in conceptually hierarchical domains. Variousembodiments of the present invention provide efficient and reliabletechniques for fusing structurally hierarchical predictions andstructurally non-hierarchical predictions. Such solutions make importanttechnical contributions to classification models in hierarchicalprediction domains, as they enable such models to utilize bothpredictive insights provided by hierarchical relationships of the outputspace and predictive insights provided without taking hierarchicalrelationships among training data into account. In doing so, variousembodiments of the present invention address key challenges related toefficiency and reliability of hierarchical predictive relationships.

FIG. 20 is a flowchart diagram of an example process 2000 for generatingpredictions based on structurally hierarchical predictions andstructurally non-hierarchically predictions. Via the varioussteps/operations of process 2000, the structured fusion unit 113 of theclassification computing entity 106 can fuse structurally hierarchicalpredictions and structurally non-hierarchical predictions in order toutilize both predictive insights provided by hierarchical relationshipsof the output space and predictive insights provided without takinghierarchical relationships among training data into account in makingpredictive inferences.

The process 2000 begins at step/operation 2001 when the structuredfusion unit 113 obtains K structurally hierarchical predictions and Mstructurally non-hierarchical predictions, where each of K and M may beone or more. In some embodiments, at least some of the K structurallyhierarchical predictions are generated by an online ML model. In someembodiments, at least some of the M structurally non-hierarchicalpredictions are generated by a con-occurrence analysis model. In someembodiments, at least some of the M structurally non-hierarchicalpredictions are generated by a ML model that utilizes at least onenatural language processing algorithm. In some embodiments, K isdetermined based on at least one of a preconfigured parameter and/orhyper-parameter of an online ML model, a preconfigured parameter and/orhyper-parameter of a particular training algorithm utilized by the modelgeneration unit 115 to train an online ML model, a parameter and/orhyper-parameter of an online ML model determined by using a MLalgorithm, a parameter and/or hyper-parameter of a particular trainingalgorithm utilized by the model generation unit 115 to train an onlineML model which is determined by using a ML algorithm, and/or the like.In some embodiments, M is determined based on at least one of thefollowing: a preconfigured parameter and/or hyper-parameter of aco-occurrence analysis ML model, a preconfigured parameter and/orhyper-parameter of a natural-language-processing-based model, apreconfigured parameter and/or hyper-parameter of a particular trainingalgorithm utilized by the model generation unit 115 to train aco-occurrence analysis ML model, a preconfigured parameter and/orhyper-parameter of a particular training algorithm utilized by the modelgeneration unit 115 to train a natural-language-processing-based model,a parameter and/or hyper-parameter of a co-occurrence analysis ML modeldetermined by using a ML algorithm, a parameter and/or hyper-parameterof a natural-language-processing-based model determined by using a MLalgorithm, a parameter and/or hyper-parameter of a particular trainingalgorithm utilized by the model generation unit 115 to train aco-occurrence analysis ML model which is determined by using a MLalgorithm, a parameter and/or hyper-parameter of a particular trainingalgorithm utilized by the model generation unit 115 to train anatural-language-processing-based model which is determined by using aML algorithm, and/or the like.

FIG. 21 provides an operational example of a structurally hierarchicalprediction set 2100. The example structurally hierarchical predictionset 2100 includes five structurally hierarchical predictions generatedby an FTRL ML model, each of which belongs to a relativelymost-dependent prediction node in a hierarchical prediction domainassociated with the structurally hierarchical prediction set 2100. Thefive structurally hierarchical predictions in the structurallyhierarchical prediction set 2100 include: structurally hierarchicalprediction 2101 corresponding to the prediction label HPO_0008;structurally hierarchical prediction 2102 corresponding to theprediction label HPO_0006; structurally hierarchical prediction 2103corresponding to the prediction label HPO_0001; structurallyhierarchical prediction 2104 corresponding to the prediction labelHPO_0002; and structurally hierarchical prediction 2105 corresponding tothe prediction label HPO_0004.

In some embodiments, the order of the structurally hierarchicalpredictions in the structurally hierarchical prediction set 2100indicates an order of magnitudes of the respective structurallyhierarchical predictions in the structurally hierarchical prediction set2100. In some embodiments, the order of the structurally hierarchicalpredictions in the structurally hierarchical prediction set 2100indicates an order of the hierarchical level the structurallyhierarchical predictions in the structurally hierarchical prediction set2100 starting from the most-dependent hierarchical level. In someembodiments, the order of the structurally hierarchical predictions inthe structurally hierarchical prediction set 2100 indicates both anorder of magnitudes of the respective structurally hierarchicalpredictions in the structurally hierarchical prediction set 2100 and anorder of the hierarchical level the structurally hierarchicalpredictions in the structurally hierarchical prediction set 2100starting from the most-dependent hierarchical level, such that thestructurally hierarchical predictions are first ranked by thehierarchical level and then structurally hierarchical predictionsbelonging to the same hierarchical level are ranked by the magnitude ofprediction scores for the respective structurally hierarchicalpredictions belonging to the same hierarchical level.

FIG. 22 provides an operational example of a structurallynon-hierarchical prediction set 2200. The example structurallynon-hierarchical prediction set 2200 includes five structurallyhierarchical predictions generated by a co-occurrence analysis ML model,each of which may belong to any hierarchical level in a hierarchicalprediction domain associated with the structurally non-hierarchicalprediction set 2200. The five structurally hierarchical predictions inthe structurally non-hierarchical prediction set 2200 include:structurally non-hierarchical prediction 2201 corresponding to theprediction label HPO_0002; structurally non-hierarchical prediction 2202corresponding to the prediction label HPO_0021; structurallynon-hierarchical prediction 2203 corresponding to the prediction labelHPO_0211; structurally non-hierarchical prediction 2204 corresponding tothe prediction label HPO_0003; and structurally non-hierarchicalprediction 2205 corresponding to the prediction label HPO_0005. In someembodiments, the order of the structurally non-hierarchical predictionsin the structurally non-hierarchical prediction set 2200 indicates anorder of magnitudes of the respective structurally non-hierarchicalpredictions in the structurally non-hierarchical prediction set 2200.

Returning to FIG. 20, at step/operation 2002, the structured fusion unit113 determines an up-weighting score for each structurally hierarchicalprediction among the K structurally hierarchical predictions obtained instep/operation 2001. In some embodiments, to determine an up-weightingscore for a particular structurally hierarchical prediction obtained instep/operation 2001, the structured fusion unit 113 determines asmallest degree of hierarchical separation between the particularstructurally hierarchical prediction and a structurally non-hierarchicalprediction among the M structurally non-hierarchical predictionsobtained in step/operation 2002. The structured fusion unit 113 thendetermines the up-weighting score for the particular structurallyhierarchical-prediction based on the determined smallest degree ofhierarchical separation between the particular structurally hierarchicalprediction and a structurally non-hierarchical prediction among the Mstructurally non-hierarchical predictions obtained in step/operation2002.

For example, to determine an up-weighting score for a particularstructurally hierarchical prediction obtained in step/operation 2001,the structured fusion unit 113 may first determine whether theparticular structurally hierarchical prediction is among the Mstructurally non-hierarchical predictions obtained in step/operation2002. If the structured fusion unit 113 determines that the particularstructurally hierarchical prediction is among the M structurallynon-hierarchical predictions obtained in step/operation 2002, thestructured fusion unit 113 determines a highest possible up-weightingscore J for the particular structurally hierarchical prediction. In someembodiments, if the structured fusion unit 113 determines that theparticular structurally hierarchical prediction is not among the Mstructurally non-hierarchical predictions obtained in step/operation2002, the structured fusion unit 113 determines if a parent of theparticular structurally hierarchical prediction up to the Jth parent ofthe particular structurally hierarchical prediction is among the Mstructurally non-hierarchical predictions obtained in step/operation2002. If the structured fusion unit 113 determines that a Dth parent ofthe particular structurally hierarchical prediction is among the Mstructurally non-hierarchical predictions obtained in step/operation2002, where D<=J, the structured fusion unit 113 determines anup-weighting score of J−D for the particular structurally hierarchicalprediction. However, if the structured fusion unit 113 determines that aDth parent of the particular structurally hierarchical prediction is notamong the M structurally non-hierarchical predictions obtained instep/operation 2002, where D<=J, the structured fusion unit 113determines a smallest possible up-weighting score (e.g., an up-weightingscore of zero) for the particular structurally hierarchical prediction.In some embodiments, if the structured fusion unit 113 determines that aDth parent of the particular structurally hierarchical prediction isamong the M structurally non-hierarchical predictions obtained instep/operation 2002, regardless of whether D<=J, the structured fusionunit 113 determines an up-weighting score of J−D for the particularstructurally hierarchical prediction, a technique that may generate downweighing scores (i.e., negative up-weighting scores) for somestructurally hierarchical predictions.

In some embodiments, step/operation 2002 may be described in referenceto FIG. 23, which is an operational example of an up-weighting scoregeneration data structure 2300. The example up-weighting scoregeneration data structure 2300 determines a ranked list of structurallyhierarchical predictions, whose ranking has been determined based on theranking of the structurally hierarchical prediction set 2100 of FIG. 21.For each entry in the up-weighting score generation data structure 2300corresponding to a particular structurally hierarchical prediction, anup-weighting score is calculated in the fourth column based on whetherthe particular structurally hierarchical prediction appears in thestructurally non-hierarchical prediction set 2200 of FIG. 22, whether aparent prediction label of the particular structurally hierarchicalprediction appears in the structurally non-hierarchical prediction set2200 of FIG. 22, whether a grandparent prediction label of theparticular structurally hierarchical prediction appears in thestructurally non-hierarchical prediction set 2200 of FIG. 22, whether agreat-grandparent prediction label of the particular structurallyhierarchical prediction appears in the structurally non-hierarchicalprediction set 2200 of FIG. 22, or whether a great-great-grandparentprediction label of the particular structurally hierarchical predictionappears in the structurally non-hierarchical prediction set 2200 of FIG.22.

For example, in the up-weighting score generation data structure 2300 ofFIG. 23, because the structurally non-hierarchical prediction 2203 is agrandparent of the structurally hierarchical prediction HPO_0006, andbecause the grandparent has a degree of separation of two from thestructurally hierarchical prediction HPO_0006, the up-weighting score2301 of 4−2=2 is calculated for the structurally hierarchical predictionHPO_0006. As another example, because prediction label 2104 iscorresponds to the structurally hierarchical prediction HPO_0002, andbecause the structurally hierarchical prediction HPO_0002 has a degreeof separation of zero from itself, the up-weighting score 2302 of 4−0=4is calculated for the structurally hierarchical prediction HPO_0002. Asa further example, because prediction label 2202 is a parent of thestructurally hierarchical prediction HPO_0004, and because the parenthas a degree of separation of one from the structurally hierarchicalprediction HPO_0004, the up-weighting score 2303 of 4−1=2 is calculatedfor the structurally hierarchical prediction HPO_0004.

Returning to FIG. 20, at step/operation 2003, the structured fusion unit113 applies the up-weighting factors determined in step/operation 2002to the structurally hierarchical predictions obtained in step/operation2001 to generate an up-weighted prediction score for each structurallyhierarchical prediction. In some embodiments, to generate the up-weighedprediction scores for the structurally hierarchical predictions obtainedin step/operation 2001, the structured fusion unit 113 increases apredictive score of each structurally hierarchical prediction (e.g., apredictive score determined using an online ML model) based on theup-weighting score for the respective structurally hierarchicalprediction. In some embodiments, to generate the up-weighed predictionscores for the structurally hierarchical predictions obtained instep/operation 2001, the structured fusion unit 113 selects thestructurally hierarchical predictions based on a selection order and,for each selected structurally hierarchical prediction, increases therank of the structurally hierarchical prediction in a ranked list ofstructurally hierarchical predictions by the up-weighting score for therespective structurally hierarchical prediction to generate an updatedranked list of the structurally hierarchical predictions. In someembodiments, the selection order may be a randomized order of thestructurally hierarchical predictions and/or an order determined basedon one or more properties of the structurally hierarchical predictions.In some embodiments, after increasing the rank of the last structurallyhierarchical prediction in the selection order of the structurallyhierarchical prediction, the structured fusion unit 2001 generated theup-weighted prediction score for each structurally hierarchicalprediction based on a ranked position of the respective structurallyhierarchical prediction in the final updated ranking of the structurallyhierarchical prediction resulting from adjusting the rank of the laststructurally hierarchical prediction in the selection order for thestructurally hierarchical predictions.

In some embodiments, step/operation 2003 can be described with referenceto FIG. 24, which is an operational example of an up-weightingadjustment data structure 2400. The example up-weighting adjustment datastructure 2400 includes the structurally hierarchical predictions fromthe structurally hierarchical prediction set 2100 of FIG. 21. However,the order of the structurally hierarchical predictions has been modifiedin accordance with the up-weighting scores for the structurallyhierarchical predictions determined using the up-weighting scoregeneration data structure 2300 of FIG. 23. For example, prediction labelHPO_0002 which had a fourth-place ranking in the structurallyhierarchical prediction set 2100 of FIG. 21 has been moved up fourranking positions to the first position in the up-weighting adjustmentdata structure 2400 based on the up-weighting score of 4 for theprediction label determined using the up-weighting score generation datastructure 2300. As another example, prediction label HPO_0004 which hada fifth-place ranking in the structurally hierarchical prediction set2100 of FIG. 21 has been moved up one ranking position to the fourthposition in the up-weighting adjustment data structure 2400 based on theup-weighting score of 3 for the prediction label determined using theup-weighting score generation data structure 2300.

E. Fusion of Structure-Based Predictions and Non-Structure-BasedPredictions

Various embodiments of the present invention are directed toclassification in a hierarchical prediction domain by using at least oneof structured input data and unstructured data. Structured data mayrefer to data that can be divided into semantically-defined data objectsbased on a predefined format of the data. Examples of structured datainclude data defined using a Structured Query Language (SQL), datadefined using a file format language (such as the JavaScript ObjectNotation (JSON) language, a Comma-Separated Value (CSV) language, or anExtensible Markup Language (XML) language), and/or the like. In thehealthcare context, structured data may include medical claims data,which may include information associated with each medical claim (e.g.,information about time of a medical operation associated with a medicalclaim, one or more operation codes associated with a medical claim, costof a medical operation associated with a medical, and/or the like.) in astructured format. Unstructured data may refer to data that cannot bedivided into semantically-defined data objects based on a predefinedformat of the data. Examples of unstructured data include various typesof natural language data, such as medical notes data which includesmedical notes provided by a medical provider. Although various solutionsexist for inferring semantically-defined data objects from unstructureddata (such as various natural language processing solutions), such aninference is distinct from a straightforward division of structured datainto semantically-defined data objects based on a predefined format ofthe structured data.

Both structured data and unstructured data provide valuable predictiveinsights for predictive analysis tasks, e.g., for predictive analysistasks related to hierarchical prediction domains. For example,structured data can provide important insights about statisticaldistribution of features and prediction labels as well as sequentialchange of correlations between features and prediction labels over time.In some cases, structured data can provide insights that are out of thereach of semantically-unsophisticated and/or primarily-lexical naturallanguage processing algorithms for processing structured data. On theother hand, when properly analyzed (e.g., when analyzed usingsemantically-sophisticated synonym-based natural language processingalgorithms), unstructured data can provide a strong source of predictiveinsights about a predictive task associated with a hierarchicalprediction domain.

Despite the complimentary utility of structured data and unstructureddata in providing predictive insights relevant to classification inhierarchical prediction domains, the problem of efficiently andeffectively integrating predictions derived from structured data (i.e.,structure-based predictions) and predictions derived from unstructureddata (i.e., non-structure-based predictions) is a non-trivial problemfrom a technical standpoint. Indeed, many conventional classificationsolutions fail to efficiently and reliably integrate structure-basedpredictions and non-structure-based predictions to generate predictiveoutputs. For example, a naive combination of particular structure-basedpredictions and non-structure-based predictions fails to properlyappreciate the reciprocal implications of structure-based predictionsand non-structure-based predictions for improving models utilized togenerate each other. Indeed, one innovative aspect of the presentinvention relates to techniques for efficiently and reliably integratingstructure-based predictions and non-structure-based predictions in amanner that causes at least one of the noted sets of predictions toprovide feedback to a model utilized to generate the other.

Accordingly, various embodiments of the present invention addresstechnical challenges related to efficient and reliable fusion ofstructure-based predictions and non-structure-based predictions byutilizing at least one of the noted sets of predictions to providefeedback to a model utilized to generate the other. For example, in someembodiments, non-structure-based predictions are used as ground-truthdata to retrain one or more ML models utilized to generatestructure-based predictions, e.g., one or more of an online ML model, aco-occurrence analysis ML model, and a structured fusion ML model.Through this and similar techniques, various embodiments of the presentinvention enable feedback-loop mechanism relationships betweenstructure-based predictions and non-structure-based predictions whichserve to render the models utilized to generate at least one of thestructure-based predictions and the non-structure-based predictions moreefficient (both in terms of training efficiency and in terms ofinference efficiency) as well as more reliable. Thus, by utilizing atleast one of the noted sets of predictions to provide feedback to amodel utilized to generate the other, various embodiments of the presentinvention address technical challenges related to efficient and reliablefusion of structure-based predictions and non-structure-basedpredictions and make substantial technical improvements to conventionalsolutions for classification, such as conventional solutions forclassification in hierarchical prediction domains.

FIG. 25 provides a flowchart diagram of an example process 2500 forperforming an unstructured fusion of structure-based predictionsgenerated using structured data and non-structure-based predictionsgenerated using unstructured data. Via the various steps/operations ofthe process 2500, the unstructured fusion unit 114 of the classificationcomputing entity 106 can perform unstructured fusions in a manner thatenables feedback-loop mechanism relationships between structure-basedpredictions and non-structure-based predictions, which in turn serves torender the ML models utilized to generate at least one of thestructure-based predictions and the non-structure-based predictions moreefficient.

The process 2500 begins at step/operation 2501 when the unstructuredfusion unit 114 obtains non-structure-based predictions for a predictiveentity. In some embodiments, the unstructured fusion unit 114 receivesat least some of the non-structure-based predictions from an externalcomputing entity 102, such as an external computing entity 102associated with a healthcare delivery organization, an externalcomputing entity 102 associated with a health insurance providerorganization, an external computing entity 102 associated with anauditing organization, an external computing entity 102 associated witha regulatory organization, and/or the like. In some embodiments, theunstructured fusion unit 114 retrieves at least some of thenon-structure-based predictions from the storage subsystem 108, e.g.,from the unstructured input data 122 stored on the storage subsystem108.

In some embodiments, the unstructured fusion unit 114 generates at leastsome of the non-structure-based predictions, e.g., by applying one ormore natural language processing algorithms to one or more unstructuredprediction inputs such as one or more synonym-based natural languageprocessing algorithms. In some embodiments, to generate at least some ofthe non-structure-based predictions as part of the step/operation 2501,the structured fusion unit 113 performs the various steps/operations ofFIG. 26, which is a flowchart diagram of an example process forgenerating structurally non-hierarchical predictions based onunstructured prediction inputs.

The example process depicted in FIG. 26 begins at step/operation 2601when the unstructured fusion unit 114 processes unstructured input dataentries to extract noun chunks from the unstructured input data entries.In some embodiments, the unstructured fusion unit 114 identifies allnoun chunks in the unstructured input data entries based on presence ofone or more separator characters (e.g., whitespace character, the dotcharacter, the comma character, and/or the like.) in the unstructuredinput data entries. In some embodiments, the unstructured fusion unit114 performs a search of the unstructured input data entries todetermine instances of occurrences of particular preconfigured termsamong the unstructured input data entries. The particular preconfiguredterms may be received from an external computing entity 102 and/or fromthe model definition data 123 for the unstructured fusion ML modelutilized by the unstructured fusion unit 114. In some embodiments, theunstructured fusion unit 114 identifies all noun chunks in theunstructured input data entries and subsequently determines which of thenoun chunks correspond to preconfigured terms.

In some embodiments, step/operation 2601 may be described with referenceto aspects of FIG. 27, which provides an operational example of anunstructured input data object 2700. The example an unstructured inputdata object 2700 depicted in FIG. 27 includes a bullet list entry 2701.In the bullet list entry 2701, the unstructured fusion unit 114 hasidentified the following preconfigured terms: the preconfigured term2711 (“renal dysplasia”), the preconfigured term 2712 (“rapidprogression”), and the preconfigured term 2713 (“hearing loss”). In someembodiments, terms such as renal dysplasia, rapid progression, andhearing loss may be deemed to have semantic significance based on modeldefinition data 123 associated with the unstructured fusion ML modelutilized by the unstructured fusion unit 114.

Returning to FIG. 26, at step/operation 2602, the unstructured fusionunit 114 maps each extracted noun chunk generated in step/operation 2601to a vector space. In some embodiments, the unstructured fusion unit 114identifies a vector space having 1 dimensions, where each of the ldimensions is associated with one or more features. The unstructuredfusion unit 114 generates, for each particular dimension of the ldimensions and each particular extracted noun, one or more particularfeature values for the particular extracted noun chunks which correspondto the one or more feature values for the particular dimension. Then,the unstructured fusion unit 114 determines a value for the dimensionand for particular extracted noun based on the one or more particularfeature values for the particular extracted noun chunks which correspondto the one or more feature values for the particular dimension. In someembodiments, at least some of the l dimensions of the vector spacediscussed above and/or at least some of the dimension values forparticular noun chunks are determined using one or more featureembedding algorithms. In some embodiments, at least some of the ldimensions of the vector space discussed above are determined based ontwo or more dimensionally-reduced features and/or by utilizing at leastone dimensionality reduction process.

In some embodiments, aspects of the step/operation 2602 may be performedin accordance with aspects of the unstructured input data object 2700which includes, in addition to the bullet list entry 2701, anunstructured prediction entry 2721-2723 for each extracted noun-chunk2711-2713. Each unstructured prediction entry 2721-2723 corresponding toan extracted noun-chunk 2711-2713 includes a feature vector for theextracted noun-chunk 2711-2713 which includes the dimension values forthe extracted noun-chunk 2711-2713 in a vector space associated with theextracted noun-chunks 2711-2713. For example, as indicated in theunstructured prediction entry 2721, the extracted noun chunk 2711(associated with the term “renal dysplasia”) is associated with thefeature vector [27::42]. As another example, as indicated in theunstructured prediction entry 2722, the extracted noun chunk 2712(associated with the term “rapid progression”) is associated with thefeature vector [150::167]. As a further example, as indicated in theunstructured prediction entry 2723, the extracted noun chunk 2713(associated with the term “hearing loss”) is associated with the featurevector [171::183].

The feature vectors generated for each extracted noun chunk generated instep/operation 2601 are then used to determine a position of eachextracted noun chunk in a vector space. FIG. 28 provides an operationalexample of a non-structure-based prediction vector space 2800 havingthree dimensions 2801, 2802, and 2803 as well as two points 2811 and2812. In the non-structure-based prediction vector space 2800 of FIG.28, the point 2811 indicates position of the term “rental dysplasia” inrelation to each of the three dimensions 2801, 2802, and 2803.Furthermore, the point 2811 indicates position of the term“branchio-ato-rental dysplasia” in relation to each of the threedimensions 2801, 2802, and 2803.

Returning to FIG. 26, at step/operation 2603, the unstructured fusionunit 114 generates non-structure-based predictions based on the vectorspace generated in step/operation 2602. In some embodiments, theunstructured fusion unit 114 applies one or more natural languageprocessing algorithms to vector space values associated with theextracted noun-chunks to determine non-structure-based predictions basedon the vector space. For example, the unstructured fusion unit 114 maydetermine HPO prediction label as non-structure-based predictions basedon the vector space. Examples of HPO prediction labels generated asnon-structure-based predictions are depicted in the unstructuredprediction entries 2721-2723 of the unstructured input data object 2700of FIG. 27. As depicted in the unstructured input data object 2700 ofFIG. 27, the unstructured prediction entry 2721 includes the HPOprediction label HPO_0000110 for the preconfigured term 2711 (associatedwith the term “rental dysplasia”); the unstructured prediction entry2722 includes the HPO prediction label HPO_0003678 for the preconfiguredterm 2712 (associated with the term “rapid progression”); and theunstructured prediction entry 2723 includes the HPO prediction labelHPO_0000365 for the preconfigured term 2713 (associated with the term“hearing loss”).

Returning to FIG. 25, at step/operation 2502, the unstructured fusionunit 114 uses the non-structure-based predictions obtained instep/operation 2501 as ground-truth prediction labels to retrain atleast one structure-based ML model used to generate structure-basedpredictions and generate new structure-based predictions. In someembodiments, the unstructured fusion unit 114 retrains some or all ofthe structure-based ML models used to generate structure-basedpredictions. Then, the unstructured fusion unit 114 uses the retrainedstructure-based models to generate new structure-based predictions.Aspects of the step/operation 1502 provide a feedback-loop mechanismwhere non-structure-based predictions are used to optimize and improvestructure-based ML models. As discussed above and further discussedbelow, this addresses a major technical challenge associated withunstructured fusion of structure-based predictions andnon-structure-based predictions.

Despite the complimentary utility of structured data and unstructureddata in providing predictive insights relevant to classification inhierarchical prediction domains, the problem of efficiently andeffectively integrating predictions derived from structured data (i.e.,structure-based predictions) and predictions derived from unstructureddata (i.e., non-structure-based predictions) is a non-trivial problemfrom a technical standpoint. Indeed, many conventional classificationsolutions fail to efficiently and reliably integrate structure-basedpredictions and non-structure-based predictions to generate predictiveoutputs. For example, a naive combination of particular structure-basedpredictions and non-structure-based predictions fails to properlyappreciate the reciprocal implications of structure-based predictionsand non-structure-based predictions for improving models utilized togenerate each other. Indeed, one innovative aspect of the presentinvention relates to techniques for efficiently and reliably integratingstructure-based predictions and non-structure-based predictions in amanner that causes at least one of the noted sets of predictions toprovide feedback to a model utilized to generate the other.

Accordingly, various embodiments of the present invention addresstechnical challenges related to efficient and reliable fusion ofstructure-based predictions and non-structure-based predictions byutilizing at least one of the noted sets of predictions to providefeedback to a model utilized to generate the other. For example, in someembodiments, non-structure-based predictions are used as ground-truthdata to retrain one or more ML models utilized to generatestructure-based predictions, e.g., one or more of an online ML model, aco-occurrence analysis ML model, and a structured fusion ML model.Through this and similar techniques, various embodiments of the presentinvention enable feedback-loop mechanism relationships betweenstructure-based predictions and non-structure-based predictions whichserve to render the models utilized to generate at least one of thestructure-based predictions and the non-structure-based predictions moreefficient (both in terms of training efficiency and in terms ofinference efficiency) as well as more reliable. Thus, by utilizing atleast one of the noted sets of predictions to provide feedback to amodel utilized to generate the other, various embodiments of the presentinvention address technical challenges related to efficient and reliablefusion of structure-based predictions and non-structure-basedpredictions and make substantial technical improvements to conventionalsolutions for classification, such as conventional solutions forclassification in hierarchical prediction domains.

At step/operation 2503, the unstructured fusion unit 114 generatesunstructured-fused predictions based on the non-structure-basedpredictions obtained in step/operation 2501 and the structure-basedpredictions generated in step/operation 2502. In some embodiments, theunstructured fusion unit 114 combines at least a portion of thenon-structure-based predictions obtained in step/operation 2501 and atleast a portion of the structure-based predictions generated instep/operation 2502 to generate unstructured-fused predictions. In someembodiments, step/operation 2503 can be performed in accordance withFIG. 29, which is a flowchart diagram of an example process forgenerating unstructured-fused predictions. The example process depictedin FIG. 29 begins at step/operation 2901 when the unstructured fusionunit 114 obtains structure-based predictions. At step/operation 2902,the unstructured fusion unit 114 obtains structure-based predictions. Atstep/operation 2902, the unstructured fusion unit 114 obtainsnon-structure-based predictions. At step/operation 2903, theunstructured fusion unit 114 selects all of the non-structure-basedpredictions and top V structure-based predictions having the highestprediction score and/or prediction rank as the unstructured fusedpredictions. In some embodiments, the selection of all of thenon-structure-based predictions as unstructured fused predictions may bebased on an assumption of reliability of natural language processingalgorithms utilized to generate the mentioned non-structure-basedpredictions, e.g., reliability of synonym-based natural languageprocessing algorithms utilized to generate the mentionednon-structure-based predictions.

In some embodiments, V is a structure-based selection capacity parameterfor the unstructured fusion ML model. In some embodiments, thestructure-based selection capacity parameter is determined based on apreconfigured parameter and/or hyper-parameter of the unstructuredfusion ML model, a preconfigured parameter and/or hyper-parameter of aparticular training algorithm utilized by the model generation unit 115to train the unstructured fusion ML model, a parameter and/orhyper-parameter of the unstructured fusion ML model determined by usinga ML algorithm, a parameter and/or hyper-parameter of a particulartraining algorithm utilized by the model generation unit 115 to trainthe unstructured fusion ML model which is determined by using a MLalgorithm, and/or the like. In some embodiments, the unstructured-fusedpredictions may retrieve the structure-based selection capacityparameter for the unstructured fusion ML model as part of the modeldefinition data 123 for the unstructured fusion ML model.

F. HPO Label Prediction

HPO label prediction is an example of a prediction task related to ahierarchical prediction domain. As discussed above and further describedbelow, the hierarchical prediction domains such as the HPO label domainpresent significant problems for various classification approaches.Examples of these challenges include challenges associated withstructural complexity of the output space of such hierarchicalprediction domains as well as challenges associated with complexity ofinput space of hierarchical prediction domains. Accordingly, to performHPO label prediction using structured medical data and unstructuredmedical data, there is a need for predictive analysis solutions thataddress the complexities associated with the HPO label space as well asthe complexities associated with processing both structured medical dataand unstructured medical data.

To perform predictions in a hierarchical prediction domains usingstructured input data and/or unstructured input data, variousembodiments of the present invention propose various arrangements of oneor more of the following ML models: an online ML model for processingstructured input data to generate structure-based predictions, aco-occurrence analysis ML model for processing structured input data togenerate structure-based predictions, a structured fusion ML model forcombining structure-based predictions, and an unstructured fusion MLmodel for combining structure-based predictions and non-structure-basedpredictions. In some embodiments, at least two of the mentioned MLmodels are organized in an ensemble architecture to generate a finalprediction based on predictions of the at least two ML models. In someembodiments, all of the mentioned ML models are organized in an ensemblearchitecture to generate a final prediction based on predictions of theat least two ML models. Such ensemble architectures provide efficientand reliable solutions for classification in hierarchical predictiondomain, such as for HPO label prediction in relation to the HPO labeldomain.

In addition, hierarchical prediction domains like the HPO domain presentunique challenges for online learning algorithms. When utilized togenerate predictions related to hierarchical prediction domains, onlinelearning algorithms should accommodate hierarchical predictiverelationships between various prediction nodes in determining how tointerpret incoming training data. Without applying appropriateoperational adjustments that address hierarchical nature of a relevantprediction domain, online learning algorithms will require higheramounts of training data, will take longer to train, and will oncetrained be less accurate and reliable. Because of those challenges,various existing online learning algorithms are ill-suited forefficiently and reliably performing classification in relation tohierarchical prediction domains.

Various embodiments of the present invention address efficiency andreliability challenges related to utilizing online learning algorithmsto generate predictions related to hierarchical prediction domains.According to one aspect that relates to improving efficiency andreliability of online learning in hierarchical prediction domains,various embodiments of the present invention eliminate a bias term usedto penalize lack of selection of a prediction node, as hierarchicalpredictive relationships complicate implications of such a lack ofselection for adjusting model parameters. For example, selection of aprediction node may have different implications for prediction nodesthat are dependent on the particular prediction node, prediction nodesfrom which the particular prediction node depends, and other predictionnodes without hierarchical relationships with the particular predictionnode. To address such complications, various embodiments of the presentinvention will not penalize lack of selection of a particular node whenadjusting parameters of a relevant ML model. In doing so, variousembodiments of the present invention address efficiency and reliabilitychallenges related to utilizing online learning algorithms to generatepredictions related to hierarchical prediction domains, such generatingHPO label predictions related to the HPO label domain.

Next, some aspects of the co-occurrence analysis ML models describedherein include important contributions to efficiency and reliability ofML in hierarchical prediction domains, such as the HPO predictiondomain. In hierarchical prediction domains, the presence of hierarchicalrelationships between prediction nodes in the output space complicatesthe task of inferring a prediction output based on prediction scores forvarious prediction nodes. On the one hand, the hierarchicalrelationships between prediction nodes in the output space provideimportant domain information that can facilitate efficient and reliablepredictive inferences. On the other hand, important predictiveconclusions may be inferred from ignoring the hierarchicalrelationships, especially in instances where the available hierarchicalmodels do not capture all of the relevant information about conceptualrelationships between prediction nodes and/or include potentiallyerroneous information about conceptual relationships between predictionnodes. Thus, there is a continuing technical challenge associated withperforming predictive analyses in a manner that takes into account bothhierarchical composition of the output space and cross-hierarchicalcomposition of the output space.

Various embodiments of the present invention address the mentionedtechnical challenges associated with considering both hierarchicalcomposition of the output space and cross-hierarchical composition ofthe output space when performing classification in hierarchicalprediction domains. For example, various embodiments of the presentinvention relate to co-occurrence analysis ML models that generatestructurally non-hierarchical predictions. A structurallynon-hierarchical prediction may be a prediction determined withoutregard to a position of the corresponding prediction node in astructural hierarchy characterizing the hierarchical prediction domainthat includes the corresponding prediction node. By generatingstructurally non-hierarchical predictions, various embodiments of thepresent invention provide predictions that are agnostic to thehierarchical composition of the prediction output space. When used incombination and/or in fusion with structurally hierarchical predictions(e.g., online learning predictions generated by an online ML model),such predictions can provide important cross-hierarchical conceptualinferences that can in turn facilitate efficient and effectiveclassification in conceptually hierarchical domains. Thus, by generatingstructurally non-hierarchical predictions that can in turn be used incombination and/or in fusion with structurally hierarchical predictions,various embodiments of the present invention address technicalchallenges related to accounting for both hierarchical composition ofthe output space and cross-hierarchical composition of the output spacewhen performing classification in hierarchical prediction domains. Indoing so, various embodiments of the present invention make importanttechnical contributions to efficiency and reliability of classificationin hierarchical prediction domains, such as in classification in an HPOprediction domain and with respect to the HPO label predictionpredictive task.

Furthermore, hierarchical prediction domains like the HPO predictiondomain present challenges related to fusion of structurally hierarchicalpredictions and non-structurally hierarchical predictions. By generatingstructurally non-hierarchical predictions, various embodiments of thepresent invention provide predictions that are agnostic to thehierarchical composition of the prediction output space. Suchstructurally non-hierarchical predictions can in turn be used incombination and/or in fusion with structurally hierarchical predictions,such as structurally hierarchical predictions generated by an onlinelearning unit 111. When structurally non-hierarchical predictions areused in combination and/or in fusion with structurally hierarchicalpredictions are used to generate structure-based predictions, suchstructured-fused predictions can provide important cross-hierarchicalconceptual inferences that can in turn facilitate efficient andeffective classification in conceptually hierarchical domains. Variousembodiments of the present invention provide efficient and reliabletechniques for fusing structurally hierarchical predictions andstructurally non-hierarchical predictions. Such solutions make importanttechnical contributions to classification models in hierarchicalprediction domains, as they enable such models to utilize bothpredictive insights provided by hierarchical relationships of the outputspace and predictive insights provided without taking hierarchicalrelationships among training data into account. In doing so, variousembodiments of the present invention address key challenges related toefficiency and reliability of classification in hierarchical predictiondomains, such as the efficiency and reliability of HPO label prediction.

Moreover, hierarchical prediction domains like the HPO domain presentchallenges related to fusion of structure-based predictions andnon-structure-based predictions. Both structured data and unstructureddata provide valuable predictive insights for predictive analysis tasks,e.g., for predictive analysis tasks related to hierarchical predictiondomains. However, despite the complimentary utility of structured dataand unstructured data in providing predictive insights relevant toclassification in hierarchical prediction domains, the problem ofefficiently and effectively integrating predictions derived fromstructured data (i.e., structure-based predictions) and predictionsderived from unstructured data (i.e., non-structure-based predictions)is a non-trivial problem from a technical standpoint. Indeed, manyconventional classification solutions fail to efficiently and reliablyintegrate structure-based predictions and non-structure-basedpredictions to generate predictive outputs. For example, a naivecombination of particular structure-based predictions andnon-structure-based predictions fails to properly appreciate thereciprocal implications of structure-based predictions andnon-structure-based predictions for improving models utilized togenerate each other. Indeed, one innovative aspect of the presentinvention relates to techniques for efficiently and reliably integratingstructure-based predictions and non-structure-based predictions in amanner that causes at least one of the noted sets of predictions toprovide feedback to a model utilized to generate the other.

Accordingly, various embodiments of the present invention addresstechnical challenges related to efficient and reliable fusion ofstructure-based predictions and non-structure-based predictions byutilizing at least one of the noted sets of predictions to providefeedback to a model utilized to generate the other. For example, in someembodiments, non-structure-based predictions are used as ground-truthdata to retrain one or more ML models utilized to generatestructure-based predictions, e.g., one or more of an online ML model, aco-occurrence analysis ML model, and a structured fusion ML model.Through this and similar techniques, various embodiments of the presentinvention enable feedback-loop mechanism relationships betweenstructure-based predictions and non-structure-based predictions whichserve to render the models utilized to generate at least one of thestructure-based predictions and the non-structure-based predictions moreefficient (both in terms of training efficiency and in terms ofinference efficiency) as well as more reliable. Thus, by utilizing atleast one of the noted sets of predictions to provide feedback to amodel utilized to generate the other, various embodiments of the presentinvention address technical challenges related to efficient and reliablefusion of structure-based predictions and non-structure-basedpredictions and make substantial technical improvements to conventionalsolutions for classification, such as conventional solutions forclassification in hierarchical prediction domains.

FIG. 30 is a flowchart diagram of an example process 3000 for performingHPO-based predictions. Via the various steps/operations of process 3000,the system interaction unit 116 of the classification computing entity106 can perform efficient and reliable classification to generate HPOpredictions and can effectively utilize the HPO predictions to generateprecision medicine analytics.

The process 3000 starts at step/operation 3001 when the systeminteraction unit 116 obtains HPO label predictions for a number ofpatients. In some embodiments, the system interaction unit 116 generatesat least some of the HPO label prediction labels using one or more ofthe ML models discussed above, e.g., using the ensemble architecture 410depicted in FIG. 4A and/or the ensemble architecture 450 depicted inFIG. 4B. In some embodiments, at least some of the HPO label predictionlabels is obtained from an external computing entity 102, such as anexternal computing entity 102 associated with a healthcare deliveryorganization, an external computing entity 102 associated with a healthinsurance provider organization, an external computing entity 102associated with an auditing organization, an external computing entity102 associated with a regulatory organization, and/or the like. In someembodiments, at least some of the HPO label prediction labels isretrieved from a local and/or remote database, such as from the storagesubsystem 108 of the classification system 101.

At step/operation 3002, the system interaction unit 116 processes theHPO label predictions to generate a standardized genetic testingframework. In some embodiments, the system interaction unit 116 appliesa standardization model to the HPO label predictions to generate astandardized genetic testing framework, where the standardization modelmay be obtained from an external computing entity, may be retrieved fromthe model definition data 123 stored on the storage subsystem, may beretrieved from a local and/or remote database, and/or the like. In someembodiments, the system interaction unit 116 provides the standardizedgenetic testing framework to at least one external computing entity 102and/or stores the standardized genetic testing framework on the storagesubsystem 108.

At step/operation 3003, the system interaction unit 116 processes theHPO label predictions to generate an integrated genomic recordrepository. Examples of data in the integrated genomic data repositoryinclude the patient-specific medical code record 3100 of FIG. 31, thepatient-specific phenotypical record 3200 of FIG. 32, and thecross-patient holistic record 3300 of FIG. 33. In some embodiments, thesystem interaction unit 116 provides the integrated genomic recordrepository to at least one external computing entity 102 and/or storesthe integrated genomic record repository on the storage subsystem 108.At step/operation 3004, the system interaction unit 116 processes theHPO label predictions to generate precision medicine analytics. In someembodiments, the precision medicine analytics include one or morecross-patient conclusions about health patterns and/or health-relatedcorrelations among a group of patients. In some embodiments, the systeminteraction unit 116 provides the precision medicine analytics to atleast one external computing entity 102 and/or stores the precisionmedicine analytics on the storage subsystem 108. As depicted, eachcross-patient holistic record entry of the cross-patient holistic record3300 of FIG. 33 includes a binary (i.e., 0 or 1) field that denoteswhether a particular predictive entity (e.g., a particular patient) ispredicted to have a particular prediction label (e.g., a particularHPO). In some embodiments, each cross-patient holistic record entry iscolored according to whether the corresponding binary field for theparticular cross-patient holistic record denotes a positive or anegative prediction (e.g., using a green color for a positive predictionand a red color for a negative prediction).

V. CONCLUSION

Many modifications and other embodiments will come to mind to oneskilled in the art to which this disclosure pertains having the benefitof the teachings presented in the foregoing descriptions and theassociated drawings. Therefore, it is to be understood that thedisclosure is not to be limited to the specific embodiments disclosedand that modifications and other embodiments are intended to be includedwithin the scope of the appended claims. Although specific terms areemployed herein, they are used in a generic and descriptive sense onlyand not for purposes of limitation.

Although example processing systems have been described in the figuresherein, implementations of the subject matter and the functionaloperations described herein can be implemented in other types of digitalelectronic circuitry, or in computer software, firmware, or hardware,including the structures disclosed in this specification and theirstructural equivalents, or in combinations of one or more of them.

Embodiments of the subject matter and the operations described hereincan be implemented in digital electronic circuitry, or in computersoftware, firmware, or hardware, including the structures disclosed inthis specification and their structural equivalents, or in combinationsof one or more of them. Embodiments of the subject matter describedherein can be implemented as one or more computer programs, i.e., one ormore modules of computer program instructions, encoded oncomputer-readable storage medium for execution by, or to control theoperation of, information/data processing apparatus. Alternatively, orin addition, the program instructions can be encoded on anartificially-generated propagated signal, e.g., a machine-generatedelectrical, optical, or electromagnetic signal, which is generated toencode information/data for transmission to suitable receiver apparatusfor execution by an information/data processing apparatus. Acomputer-readable storage medium can be, or be included in, acomputer-readable storage device, a computer-readable storage substrate,a random or serial access memory array or device, or a combination ofone or more of them. Moreover, while a computer-readable storage mediumis not a propagated signal, a computer-readable storage medium can be asource or destination of computer program instructions encoded in anartificially-generated propagated signal. The computer-readable storagemedium can also be, or be included in, one or more separate physicalcomponents or media (e.g., multiple CDs, disks, or other storagedevices).

The operations described herein can be implemented as operationsperformed by an information/data processing apparatus oninformation/data stored on one or more computer-readable storage devicesor received from other sources.

A computing entity is an example of a data processing apparatus. Theterm “data processing apparatus” encompasses all kinds of apparatus,devices, and machines for processing data, including by way of example aprogrammable processor, a computer, a system on a chip, or multipleones, or combinations, of the foregoing. The apparatus can includespecial purpose logic circuitry, e.g., an FPGA (Field Programmable GateArray) or an ASIC (Application Specific Integrated Circuit). Theapparatus can also include, in addition to hardware, code that createsan execution environment for the computer program in question, e.g.,code that constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, a cross-platform runtimeenvironment, a virtual machine, or a combination of one or more of them.The apparatus and execution environment can realize various differentcomputing model infrastructures, such as web services, distributedcomputing and grid computing infrastructures.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, object, orother unit suitable for use in a computing environment. A computerprogram may, but need not, correspond to a file in a file system. Aprogram can be stored in a portion of a file that holds other programsor information/data (e.g., one or more scripts stored in a markuplanguage document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub-programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described herein can be performed by oneor more programmable processors executing one or more computer programsto perform actions by operating on input information/data and generatingoutput. Processors suitable for the execution of a computer programinclude, by way of example, both general and special purposemicroprocessors, and any one or more processors of any kind of digitalcomputer. Generally, a processor will receive instructions andinformation/data from a read-only memory, a random access memory, orboth. The essential elements of a computer are a processor forperforming actions in accordance with instructions and one or morememory devices for storing instructions and data. Generally, a computerwill also include, or be operatively coupled to receive information/datafrom or transfer information/data to, or both, one or more mass storagedevices for storing data, e.g., magnetic, magneto-optical disks, oroptical disks. However, a computer need not have such devices. Devicessuitable for storing computer program instructions and information/datainclude all forms of non-volatile memory, media and memory devices,including by way of example semiconductor memory devices, e.g., EPROM,EEPROM, and flash memory devices; magnetic disks, e.g., internal harddisks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROMdisks. The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described herein can be implemented on a computer having adisplay device, e.g., a CRT (Cathode Ray Tube) or LCD (Liquid CrystalDisplay) monitor, for displaying information/data to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser.

Embodiments of the subject matter described herein can be implemented ina computing system that includes a back-end component, e.g., as aninformation/data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., aclient device having a graphical user interface or a web browser throughwhich a user can interact with an implementation of the subject matterdescribed herein, or any combination of one or more such back-end,middleware, or front-end components. The components of the system can beinterconnected by any form or medium of digital information/datacommunication, e.g., a communication network. Examples of communicationnetworks include a local area network (“LAN”) and a wide area network(“WAN”), an inter-network (e.g., the Internet), and peer-to-peernetworks (e.g., ad hoc peer-to-peer networks).

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someembodiments, a server transmits information/data (e.g., a HypertextMarkup Language (HTML) page) to a client device (e.g., for purposes ofdisplaying information/data to and receiving user input from a userinteracting with the client device). Information/data generated at theclient device (e.g., a result of the user interaction) can be receivedfrom the client device at the server.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinventions or of what may be claimed, but rather as description offeatures specific to particular embodiments of particular inventions.Certain features that are described herein in the context of separateembodiments can also be implemented in combination in a singleembodiment. Conversely, various features that are described in thecontext of a single embodiment can also be implemented in multipleembodiments separately or in any suitable sub-combination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asub-combination or variation of a sub-combination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults, unless described otherwise. In certain circumstances,multitasking and parallel processing may be advantageous. Moreover, theseparation of various system components in the embodiments describedabove should not be understood as requiring such separation in allembodiments, and it should be understood that the described programcomponents and systems can generally be integrated together in a singlesoftware product or packaged into multiple software products.

Thus, particular embodiments of the subject matter have been described.Other embodiments are within the scope of the following claims. In somecases, the actions recited in the claims can be performed in a differentorder and still achieve desirable results. In addition, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults, unless described otherwise. In certain implementations,multitasking and parallel processing may be advantageous.

1. A computer-implemented method for generating a predictive outputbased at least in part on one or more structured prediction inputs andone or more structured prediction inputs, the computer-implementedcomprising: generating one or more structure-based predictions based atleast in part on the one or more structured prediction inputs and usingone or more structured machine learning models; generating one or morenon-structure-based predictions based at least in part on the one ormore unstructured prediction inputs; obtaining access to an unstructuredfusion machine learning model, wherein (i) the unstructured fusionmachine learning model is configured to perform an unstructured fusionmachine learning analysis based at least in part on the one or morestructure-based predictions and the one or more non-structure-basedpredictions to generate one or more unstructured-fused predictions; and(ii) the unstructured fusion machine learning analysis comprisesretraining the one or more structured machine learning models based atleast in part on the one or more non-structure-based predictions;performing the unstructured fusion machine learning analysis to generatethe one or more non-structure-based predictions; and generating, basedat least in part on the one or more unstructured-fused predictions, thepredictive output, wherein the predictive output indicates a selectednode of the plurality of prediction nodes for at least some of thestructurally hierarchical predictions.
 2. The computer-implementedmethod of claim 1, wherein generating the one or more structure-basedpredictions comprises: obtaining access to an online machine learningmodel, wherein (i) the online machine learning model is configured toperform an online machine learning analysis associated with ahierarchical prediction domain based at least in part on one or moreprediction inputs to generate one or more structurally hierarchicalpredictions, (ii) the hierarchical prediction domain is associated withone or more predictive hierarchical relationships among a plurality ofprediction nodes, (iii) the one or more predictive hierarchicalrelationships define, for each of the plurality of prediction nodes, acorresponding hierarchical predictive position, and (iv) eachstructurally hierarchical prediction is determined based at least inpart on the hierarchical predictive position of the correspondingprediction node; performing the online predictive machine learninganalysis based at least in part on the one or more prediction inputs togenerate the one or more structurally hierarchical predictions for theone or more prediction inputs; and generating the one or morestructure-based predictions based at least in part on the one or morestructurally hierarchical predictions in addition to the one or moreunstructured-fused predictions.
 3. The computer-implemented method ofclaim 2, wherein the online machine learning model is afollow-the-regularized leader machine learning model.
 4. Thecomputer-implemented method of claim 1, wherein generating the one ormore structure-based predictions comprises: obtaining access to aco-occurrence analysis machine learning model, wherein (i) theco-occurrence analysis machine learning model is configured to perform aco-occurrence machine learning analysis associated with a hierarchicalprediction domain based at least in part on one or more predictioninputs to generate one or more structurally non-hierarchicalpredictions, (ii) the co-occurrence analysis machine learning model isassociated with a predictive co-occurrence score between eachfeature-node pair of a predictive feature set of one or more predictivefeature sets and a prediction node of the plurality nodes, (iii) thehierarchical prediction domain is associated with one or morehierarchical predictive relationships among a plurality of predictionnodes, (iv) the one or more hierarchical predictive relationshipsdefine, for each of the plurality of prediction nodes, a correspondinghierarchical predictive position, and (v) each of the one or morestructurally non-hierarchical predictions is determined without regardto the hierarchical predictive position of the corresponding predictionnode; performing the co-occurrence machine learning analysis based atleast in part on the one or more prediction inputs to generate the oneor more structurally non-hierarchical predictions for the one or moreprediction inputs; and generating the one or more structure-basedpredictions based at least in part on the one or more structurallynon-hierarchical predictions in addition to the one or moreunstructured-fused predictions.
 5. The computer-implemented method ofclaim 4, wherein each predictive co-occurrence score for a correspondingprediction node is determined based at last in part on: a node-widenormalization of the predictive co-occurrence score based at least inpart on each raw predictive score of one or more raw predictive scoresassociated with the prediction node, and a significance-based filteringof the particular predictive co-occurrence score based at least in parton each predictive co-occurrence score between each feature-node pair.6. The computer-implemented method of claim 4, wherein each predictiveco-occurrence score for a corresponding predictive feature set isdetermined based at last in part on: a feature-wide normalization of thepredictive co-occurrence score based at least in part on each rawpredictive score of one or more raw predictive scores associated withthe prediction feature set, and a significance-based filtering of theparticular predictive co-occurrence score based at least in part on eachpredictive co-occurrence score between each feature-node pair.
 7. Thecomputer-implemented method of claim 1, wherein generating the one ormore structure-based predictions comprises: obtaining one or morestructurally hierarchical predictions for the one or more predictioninputs, wherein (i) the one or more structurally hierarchicalpredictions are associated with a predictive hierarchical relationship,(ii) the hierarchical prediction domain is associated with one or morepredictive hierarchical relationships among a plurality of predictionnodes, (iii) the one or more predictive hierarchical relationshipsdefine, for each of the plurality of prediction nodes, a correspondinghierarchical predictive position, and (iv) each structurallyhierarchical prediction is determined based at least in part on thehierarchical predictive position of the corresponding prediction node;obtaining one or more structurally non-hierarchical predictions for theone or more prediction inputs, wherein each of the one or morestructurally non-hierarchical predictions is determined without regardto the hierarchical predictive position of the corresponding predictionnode; obtaining access to an structured fusion machine learning model,wherein (i) the structured fusion machine learning model is configuredto perform an unstructured fusion machine learning analysis based atleast in part on the one or more structure-based predictions and the oneor more structure-based predictions to generate one or morenon-structure-based predictions; and (ii) the structured fusion machinelearning analysis comprises retraining one or more structured machinelearning models comprising the structured fusion machine learning modelbased at least in part on the one or more non-structure-basedpredictions; performing the structured fusion machine learning analysisto generate the one or more structure-based predictions; and generatingthe one or more structure-based predictions based at least in part onthe one or more structure-based predictions in addition to the one ormore unstructured-fused predictions.
 8. The computer-implemented methodof claim 7, wherein the structured fusion machine learning analysis isconfigured to adjust a predictive rank of each structurally hierarchicalprediction based at least in part on a hierarchical prediction distancebetween a first prediction node of the plurality of prediction nodesthat is associated with the structurally hierarchical prediction and asecond prediction node of the plurality of prediction nodes that is aparent prediction node of the first prediction node and that isassociated with a structurally non-hierarchical prediction of the one ormore structurally non-hierarchical predictions.
 9. Thecomputer-implemented method of claim 7, wherein the structured fusionmachine learning analysis is configured to adjust a predictive rank ofeach structurally hierarchical prediction based at least in part on ahierarchical prediction distance between: a first prediction node of theplurality of prediction nodes that is associated with the structurallyhierarchical prediction, and a second prediction node of the pluralityof prediction nodes that is: (i) a parent prediction node of the firstprediction node, and (ii) associated with a structurallynon-hierarchical prediction of the one or more structurallynon-hierarchical predictions.
 10. The computer-implemented method ofclaim 1, wherein: the one or more prediction inputs comprise one or moremedical feature inputs for a patient profile; the predictive outputcomprises at least one human phenotype ontology label prediction for thepatient profile; and the one or more predictive hierarchicalrelationships comprise one or more human phenotype ontology dependencyrelationships.
 11. An apparatus comprising at least one processor and atleast one non-transitory memory comprising program code, wherein the atleast one non-transitory memory and the program code are configured to,with the at least one processor, cause (Original) The apparatus at leastat least perform operations configured to at least: generate one or morestructure-based predictions based at least in part on the one or morestructured prediction inputs and using one or more structured machinelearning models; generate one or more non-structure-based predictionsbased at least in part on the one or more unstructured predictioninputs; obtain access to an unstructured fusion machine learning model,wherein (i) the unstructured fusion machine learning model is configuredto perform an unstructured fusion machine learning analysis based atleast in part on the one or more structure-based predictions and the oneor more non-structure-based predictions to generate one or moreunstructured-fused predictions; and (ii) the unstructured fusion machinelearning analysis comprises retraining the one or more structuredmachine learning models based at least in part on the one or morenon-structure-based predictions; perform the unstructured fusion machinelearning analysis to generate the one or more non-structure-basedpredictions; and generate, based at least in part on the one or moreunstructured-fused predictions, the predictive output, wherein thepredictive output indicates a selected node of the plurality ofprediction nodes for at least some of the structurally hierarchicalpredictions.
 12. The apparatus of claim 11, wherein generating the oneor more structure-based predictions comprises: obtaining access to anonline machine learning model, wherein (i) the online machine learningmodel is configured to perform an online machine learning analysisassociated with a hierarchical prediction domain based at least in parton one or more prediction inputs to generate one or more structurallyhierarchical predictions, (ii) the hierarchical prediction domain isassociated with one or more predictive hierarchical relationships amonga plurality of prediction nodes, (iii) the one or more predictivehierarchical relationships define, for each of the plurality ofprediction nodes, a corresponding hierarchical predictive position, and(iv) each structurally hierarchical prediction is determined based atleast in part on the hierarchical predictive position of thecorresponding prediction node; performing the online predictive machinelearning analysis based at least in part on the one or more predictioninputs to generate the one or more structurally hierarchical predictionsfor the one or more prediction inputs; and generating the one or morestructure-based predictions based at least in part on the one or morestructurally hierarchical predictions in addition to the one or moreunstructured-fused predictions.
 13. The apparatus of claim 11, whereingenerating the one or more structure-based predictions comprises:obtaining access to a co-occurrence analysis machine learning model,wherein (i) the co-occurrence analysis machine learning model isconfigured to perform a co-occurrence machine learning analysisassociated with a hierarchical prediction domain based at least in parton one or more prediction inputs to generate one or more structurallynon-hierarchical predictions, (ii) the co-occurrence analysis machinelearning model is associated with a predictive co-occurrence scorebetween each feature-node pair of a predictive feature set of one ormore predictive feature sets and a prediction node of the pluralitynodes, (iii) the hierarchical prediction domain is associated with oneor more hierarchical predictive relationships among a plurality ofprediction nodes, (iv) the one or more hierarchical predictiverelationships define, for each of the plurality of prediction nodes, acorresponding hierarchical predictive position, and (v) each of the oneor more structurally non-hierarchical predictions is determined withoutregard to the hierarchical predictive position of the correspondingprediction node; performing the co-occurrence machine learning analysisbased at least in part on the one or more prediction inputs to generatethe one or more structurally non-hierarchical predictions for the one ormore prediction inputs; and generating the one or more structure-basedpredictions based at least in part on the one or more structurallynon-hierarchical predictions in addition to the one or moreunstructured-fused predictions.
 14. The apparatus of claim 11, whereingenerating the one or more structure-based predictions comprises:obtaining one or more structurally hierarchical predictions for the oneor more prediction inputs, wherein (i) the one or more structurallyhierarchical predictions are associated with a predictive hierarchicalrelationship, (ii) the hierarchical prediction domain is associated withone or more predictive hierarchical relationships among a plurality ofprediction nodes, (iii) the one or more predictive hierarchicalrelationships define, for each of the plurality of prediction nodes, acorresponding hierarchical predictive position, and (iv) eachstructurally hierarchical prediction is determined based at least inpart on the hierarchical predictive position of the correspondingprediction node; obtaining one or more structurally non-hierarchicalpredictions for the one or more prediction inputs, wherein each of theone or more structurally non-hierarchical predictions is determinedwithout regard to the hierarchical predictive position of thecorresponding prediction node; obtaining access to an structured fusionmachine learning model, wherein (i) the structured fusion machinelearning model is configured to perform an unstructured fusion machinelearning analysis based at least in part on the one or morestructure-based predictions and the one or more structure-basedpredictions to generate one or more non-structure-based predictions; and(ii) the structured fusion machine learning analysis comprisesretraining one or more structured machine learning models comprising thestructured fusion machine learning model based at least in part on theone or more non-structure-based predictions; performing the structuredfusion machine learning analysis to generate the one or morestructure-based predictions; and generating the one or morestructure-based predictions based at least in part on the one or morestructure-based predictions in addition to the one or moreunstructured-fused predictions.
 15. The apparatus of claim 11, wherein:the one or more prediction inputs comprise one or more medical featureinputs for a patient profile; the predictive output comprises at leastone human phenotype ontology label prediction for the patient profile;and the one or more predictive hierarchical relationships comprise oneor more human phenotype ontology dependency relationships.
 16. Anon-transitory computer storage medium comprising instructionsconfigured to cause one or more processors to at least at least performoperations configured to at least: generate one or more structure-basedpredictions based at least in part on the one or more structuredprediction inputs and using one or more structured machine learningmodels; generate one or more non-structure-based predictions based atleast in part on the one or more unstructured prediction inputs; obtainaccess to an unstructured fusion machine learning model, wherein (i) theunstructured fusion machine learning model is configured to perform anunstructured fusion machine learning analysis based at least in part onthe one or more structure-based predictions and the one or morenon-structure-based predictions to generate one or moreunstructured-fused predictions; and (ii) the unstructured fusion machinelearning analysis comprises retraining the one or more structuredmachine learning models based at least in part on the one or morenon-structure-based predictions; perform the unstructured fusion machinelearning analysis to generate the one or more non-structure-basedpredictions; and generate, based at least in part on the one or moreunstructured-fused predictions, the predictive output, wherein thepredictive output indicates a selected node of the plurality ofprediction nodes for at least some of the structurally hierarchicalpredictions.
 17. The non-transitory computer storage medium of claim 16,wherein generating the one or more structure-based predictionscomprises: obtaining access to an online machine learning model, wherein(i) the online machine learning model is configured to perform an onlinemachine learning analysis associated with a hierarchical predictiondomain based at least in part on one or more prediction inputs togenerate one or more structurally hierarchical predictions, (ii) thehierarchical prediction domain is associated with one or more predictivehierarchical relationships among a plurality of prediction nodes, (iii)the one or more predictive hierarchical relationships define, for eachof the plurality of prediction nodes, a corresponding hierarchicalpredictive position, and (iv) each structurally hierarchical predictionis determined based at least in part on the hierarchical predictiveposition of the corresponding prediction node; performing the onlinepredictive machine learning analysis based at least in part on the oneor more prediction inputs to generate the one or more structurallyhierarchical predictions for the one or more prediction inputs; andgenerating the one or more structure-based predictions based at least inpart on the one or more structurally hierarchical predictions inaddition to the one or more unstructured-fused predictions.
 18. Thenon-transitory computer storage medium of claim 16, wherein generatingthe one or more structure-based predictions comprises: obtaining accessto a co-occurrence analysis machine learning model, wherein (i) theco-occurrence analysis machine learning model is configured to perform aco-occurrence machine learning analysis associated with a hierarchicalprediction domain based at least in part on one or more predictioninputs to generate one or more structurally non-hierarchicalpredictions, (ii) the co-occurrence analysis machine learning model isassociated with a predictive co-occurrence score between eachfeature-node pair of a predictive feature set of one or more predictivefeature sets and a prediction node of the plurality nodes, (iii) thehierarchical prediction domain is associated with one or morehierarchical predictive relationships among a plurality of predictionnodes, (iv) the one or more hierarchical predictive relationshipsdefine, for each of the plurality of prediction nodes, a correspondinghierarchical predictive position, and (v) each of the one or morestructurally non-hierarchical predictions is determined without regardto the hierarchical predictive position of the corresponding predictionnode; performing the co-occurrence machine learning analysis based atleast in part on the one or more prediction inputs to generate the oneor more structurally non-hierarchical predictions for the one or moreprediction inputs; and generating the one or more structure-basedpredictions based at least in part on the one or more structurallynon-hierarchical predictions in addition to the one or moreunstructured-fused predictions.
 19. The non-transitory computer storagemedium of claim 16, wherein generating the one or more structure-basedpredictions comprises: obtaining one or more structurally hierarchicalpredictions for the one or more prediction inputs, wherein (i) the oneor more structurally hierarchical predictions are associated with apredictive hierarchical relationship, (ii) the hierarchical predictiondomain is associated with one or more predictive hierarchicalrelationships among a plurality of prediction nodes, (iii) the one ormore predictive hierarchical relationships define, for each of theplurality of prediction nodes, a corresponding hierarchical predictiveposition, and (iv) each structurally hierarchical prediction isdetermined based at least in part on the hierarchical predictiveposition of the corresponding prediction node; obtaining one or morestructurally non-hierarchical predictions for the one or more predictioninputs, wherein each of the one or more structurally non-hierarchicalpredictions is determined without regard to the hierarchical predictiveposition of the corresponding prediction node; obtaining access to anstructured fusion machine learning model, wherein (i) the structuredfusion machine learning model is configured to perform an unstructuredfusion machine learning analysis based at least in part on the one ormore structure-based predictions and the one or more structure-basedpredictions to generate one or more non-structure-based predictions; and(ii) the structured fusion machine learning analysis comprisesretraining one or more structured machine learning models comprising thestructured fusion machine learning model based at least in part on theone or more non-structure-based predictions; performing the structuredfusion machine learning analysis to generate the one or morestructure-based predictions; and generating the one or morestructure-based predictions based at least in part on the one or morestructure-based predictions in addition to the one or moreunstructured-fused predictions.
 20. The non-transitory computer storagemedium of claim 16, wherein: the one or more prediction inputs compriseone or more medical feature inputs for a patient profile; the predictiveoutput comprises at least one human phenotype ontology label predictionfor the patient profile; and the one or more predictive hierarchicalrelationships comprise one or more human phenotype ontology dependencyrelationships.